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I.  INTRODUCTION 


Two  images  of  the  same  region  are  said  to  he  registered  when  equivalent 
geographic  points  of  the  scenes  in  the  two  images  coincide. 

Rectification  is  a  congruencing  process  by  which  the  geometry  of  the 
image  is  made  planimetric,  i.e.,  where  the  two  images  of  the  same  scene  are 
transformed  so  that  the  size  and  shape  of  any  object  on  one  image  is  the  same 
as  the  size  and  shape  of  that  object  on  the  other  image.  Image  rectification 
allows  a  direct  comparison  of  different  images  of  the  same  scene  either 
originating  from  different  sensors  or  from  the  same  sensor  but  under  different- 
conditions  (e.g.,  orientation,  season,  weather,  time  of  day). 

Raw,  unprocessed  images  cannot  be  registered  directly  because  of  differ¬ 
ences  in  viewing  conditions,  and  because  of  radiometric  and  geometrical  dis¬ 
tortions  (warping).  A  preprocessing  or  "unwarping"  of  the  image  data  before 
registration  and  information  extraction,  in  the  form  of  geometric  and 
radiometric  corrections  and  data  processing  (interpolation,  resampling, 
ref ormat t ing) ,  is  necessary.  This  process  of  rectification,  in  addition  to 
registration  and  change  detection,  is  the  subject  of  this  report. 

The  accurate  rectification/registration  of  data  is  a  key  requ 
several  applications  such  as  intelligence  gathering,  advanced  vehi 
guidance,  change  detection,  and  classification  of  remotely  sensed 

A.  STUDY  OBJECTIVES 

The  objective  of  the  study  is  to  evaluate  the  feasibility  of 
cation  system  having  pixel  (picture  element)  to  subpixel  registrat 
and  semiautomatic  operation  in  a  multisensor  environment.  Special 
tion  was  given  to  ground-based,  mobile,  compact  systems  with  near 
process i ng . 

The  purpose  of  the  first  phase  of  this  study  was  to  investiga 
general  process  of  rec t i f icat ion/regi s t ra t i on  of  remotely  sensed, 
tion  data  in  a  multisensor  environment  and  in  the  presence  of  milt 
geometric  distortions.  Specific  sensors  under  consideration  in  th 
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the  study  are  very  general  and  generic  in  nature,  covering  the  whole  spectrum 
from  optical/near  infrared  (IR)  to  middle  IR,  thermal  IR,  and  microwave, 
including  both  active  types  (sensor  providing  its  own  source  of  electro¬ 
magnetic  radiation)  and  passive  types  (sensor  using  naturally  reflected  energy 
from  the  terrain). 

In  the  second  phase  of  the  study,  the  emphasis  was  placed  on  the  use  of 
synthetic  aperture  radar  (SAR)  for  imaging  as  a  sole  sensor.  The  purpose  was 
to  study  the  specific  qualities  and  problems  of  SAR  matching  and  registration 
techniques  in  order  to  analyze  the  state  of  the  art  of  rectification 
algorithms  and  to  ptopose  some  future  tasks  in  this  area. 

B .  STUDY  APPROACH 

To  achieve  the  objectives,  a  step-by-step  approach  was  taken,  and  the 
following  tasks  have  been  accomplished: 

°  Literature  survey  of  rectification  and  registration  techniaues  in  use 
today  and  identification  of  the  state  of  the  art. 

®  Identification  and  study  of  candidate  techniques  or  a  combination  of 
several  algorithms  from  the  wealth  of  techniques  available.  This 
spectrum  ranges  from  classical  correction  methods  to  feature  matching 
to  hybrid  techniques. 

°  Evaluation  of  error  removal  capabilities  of  image -mat ch i ng  algorithms 
under  various  geometric  distortions  and  dependency  of  sensors  to 
different  errors. 

°  Study  of  ttie  effect  of  various  resampl  i  ng/ i  n  te  rpol  a  t  i  on  algorithms  on 
registration  accuracy. 

®  Detailed  study  of  map-matching  methods  specific  to  SAR  imagery. 

®  Preliminary  study  of  change  detection  techniques. 

®  Preliminary  study  and  analysis  in  the  areas  of  signal  processing  and 
artificial  intelligence  to  identify  novel  techniques  for  the  removal 
of  speckle  and  handling  of  variations  in  contrast.  The  existence  of 
these'  factors  lead  to  grossly  incorrect  matches.  (Speckle  is  a 
granular  noise  arising  from  the  random  interference  of  wavefronts 
scattered  from  an  optically  rough  surface.) 

•  Identification  of  several  future  approaches  and  techniques  which 
include  SAR  image  matching  to  other  SAR  images,  to  visible 
references,  or  to  forward  looking  infrared  (FLTR)  reference  images. 
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OVERVIEW 


A  typical  data  gathering  system  geometry  for  remote  sensing  applications 
is  depicted  in  Figure  1.  A  reference  map  is  provided  by  platform  A  with  an 
onboard  sensor.  Prior  to  the  remote  sensing  operation,  this  reference  map 
covers  a  much  larger  area  than  the  target  or  the  scene  of  interest,  and  is 
constructed  and  preprocessed  accurately  and  is  stored  in  the  computer  on  the 
ground.  A  sensed  image  is  generated  by  a  second  sensor  on  board  platform  B. 
This  sensed  image  contains  the  target  or  the  scene  of  interest  only  and  is 
contained  somewhere  within  the  boundaries  of  the  reference  map.  The  problem 
is  to  rectify  and  correct  the  sensed  image  in  such  a  way  that  accurate  regis¬ 
tration  of  all  points  within  the  sensed  image  (with  respect  to  the  reference 
map)  is  possible.  Rectification/registration  can  be  accomplished  on  a  com¬ 
puter  located  in  a  mobile  station  on  the  ground. 

This  report  covers  the  area  of  image  rectification,  registration,  and 
change  detection  for  images  generated  by  onboard  sensors.  Types  of  sensors 
under  consideration  in  the  first  phase  of  the  study  were  very  general  and 
generic  in  nature,  covering  the  whole  spectrum  from  optical  to  IR  to  microwave 
(sections  1  through  V).  In  the  second  phase  of  the  study,  specific  problems 
of  SAR  registration  and  rectification  were  surveyed  and  studied  (section  VI), 
and  new  future  tasks  were  proposed  (section  VII). 

The  conclusion  of  the  study  is  covered  in  the  first  section  of  the 
report,  along  with  the  introduction,  objectives,  and  overview. 

Section  II  describes  the  process  of  rectification  as  a  whole.  The 
sources  of  image  distortion  are  examined  and  various  bases  of  reference  data 
are  considered. 

Section  III  describes  geometrical  distortion  in  imaging  systems  due  to 
optical  and  orientation  effects.  A  design  concept  for  near-real -t ime  correc¬ 
tion  is  also  presented.  Correction  for  display  and  matching  is  considered, 
system  hardware  requirements  are  described,  and  superposition  of  multiple 
images  is  discussed. 
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In  general,  three  basic  types  of  rectification  algorithms  have  emerged 
from  this  study  and  are  categorized  into:  signal  processing-based,  artificial 
intelligence-based,  and  hybrid  techniques  which  make  the  best  use  of  the  first 
two  approaches.  Section  IV  describes  all  three  categories  of  image  matching 
as  well  as  change  detection  algorithms  for  multisensor  environments. 

In  real-time  systems  using  high  data  rates,  rectification  of  every 
picture  element  is  not  practical.  In  these  systems,  only  a  fraction  of 
samples  are  fully  corrected  geometrically,  and  remaining  samples  are  obtained 
using  interpolation ,  resampling,  or  statistical  approximation  means.  In 
section  V  an  overview  of  the  resampling  process  as  it  relates  to  registration 
and  rectification  of  onboard  sensor  imagery  is  discussed.  Various  one-  and 
two-dimensional  resampling  techniques  are  reviewed  and  compared . 

Very  little  work  is  available  on  the  matching  of  SAR  images  to  visible 
images  or  to  a  SAR  reference  map.  Most  of  the  previous  work  has  been  concen¬ 
trated  in  the  rectification  and  registration  of  visible  imagery  as  discussed 
in  sections  II  through  V.  Section  VI  of  this  report  deals  with  specific 
qualities  of  SAR  imagery  which  may  cause  serious  problems  in  accurate  image 
matching  and  investigates  the  feasibilities  of  various  rectification,  regis¬ 
tration,  and  change  detection  algorithms  applicable  to  SAR  imagery. 

Part  of  the  objective  of  the  second  phase  of  this  study  is  to  propose  and 
develop  techniques  for  registration  and  change  detection  with  SAR  images. 

These  techniques  may  include  SAR  image  matching  to  other  SAR  reference  images, 
to  visible  references,  or  to  FLIR  reference  images.  Because  the  problem  is 
generally  quite  difficult  and  little  work  has  been  previously  done  on  the 
problem,  several  possible  approaches  are  proposed  in  section  VII. 

Based  on  analysis  in  the  areas  of  signal  processing  and  artificial 
intelligence,  four  specific  tasks  are  proposed  for  image  registration/rectifi¬ 
cation  and  possible  change  detection  of  SAR  data: 

•  The  first  task  involves  registration  through  geometrical  transforma¬ 
tion  and  resampling. 

•  The  second  task,  which  requires  considerable  research  effort, 
involves  total  registration  and  change  detection  through  a  com¬ 
bination  of  signal  processing  and  artificial  intelligence  techniques. 


©  The  third  task  is  basically  a  simplified  version  of  the  second  task 
for  registration  of  SAR  images  with  a  known,  well-defined  visible 
image. 

©  The  last  task  involves  man-machine  interactive  methods  for  image 

rectification.  This  approach  to  rectif ication  allows  for  analytical 
evaluation  and  comparison  of  various  registration  algorithms  and 
their  performance  in  addition  to  the  utilization  of  a  photointer¬ 
preter's  inputs  to  the  system. 

D.  CONCLUSIONS  AND  RESULTS 

This  study  has  surveyed  the  entire  field  of  image  matching,  rectifi¬ 
cation,  and  registration,  including  techniques  that  are  applicable  to  a  wide 
range  of  sensor  wavelengths,  image  noise  levels,  image  detail,  and  image 
distortion.  An  important  conclusion  is  that  there  is  no  universally  robust 
technique  or  even  a  family  of  techniques  that  can  solve  all  rectification/ 
registration  problems.  The  optimum  solution  is  highly  dependent  on  the 
particular  parameters  of  the  system,  including:  speed;  the  number  of  pixels 
(resolution);  the  quality  of  the  reference  and  sensed  images;  the  size  of  the 
processor,  including  computing,  memory  and  input/output  capabilities;  the 
amount  of  human  interaction  allowed;  and  the  amount  of  a  priori  information 
avai lable . 

The  study  has  led  to  the  emergence  of  three  major  types  of  registration 
algorithms: 

©  Correlated  or  signal -processing-based  techniques  that  numerically 
compute  a  measure  of  similarity  between  images  on  a  pixe 1 -by-pixe 1 
basis 

o  Artificial  intelligence  (AI)  or  feature-based  techniques.  These 
methods  compute  similarity  between  features  extracted  from  the 
images.  The  features  can  include  edges,  boundaries,  and  vertices, 
and  matching  is  often  done  at  a  higher  (symbolic)  level 

©  Hybrid  techniques,  which  combine  the  first  two  techniques  in  a 
hierarchy  of  levels  to  make  the  best  use  of  both. 

In  general,  the  feature  matching  methods  work  best  on  images  with  high  signal- 

to-noise  ratio  (SNR)  regardless  of  wavelength.  Most  SAR  images  never  have  SNR 
adequate  for  a  direct  application  of  feature  matching  techniques.  The 


correlation-based  techniques  work  best  on  images  with  low  SNR,  where  they  pro¬ 
vide  some  inherent  noise  suppression.  They  will  also  fail,  however,  if  there 
are  significant  differences  in  orientation,  magnif ication,  illumination,  etc., 
between  reference  and  search  images.  Hybrid  techniques  are  needed  for  low-to- 
moderate  SNR  situations,  regardless  of  the  utilized  wavelength. 

Very  little  has  been  done  on  the  specific  problem  of  SAR  rectif ication 
and  registration.  It  is  generally  a  very  difficult  problem  that  requires  a 
great  deal  more  research  and  development.  The  many  unanswered  questions  about 
SAR  need  more  analysis,  simulation,  and  experimental  verification.  There  is 
research  at  many  levels  (up  to  the  doctoral  dissertation  level)  under  way  at 
many  organizations  to  find  out  some  of  these  answers,  but  they  do  not  come 
easily.  Some  of  the  specific  questions  and  tasks  discussed  in  this  study 
could  easily  consume  several  man-years  of  effort.  Many  of  the  most  useful 
practical  results  have  involved  extensive  software  development  taking  place 
over  a  period  of  years. 

On  a  more  positive  side,  however,  with  specific  well-defined  information 
about  a  particular  scenario  for  matching,  rectification,  and  registration,  it 
may  be  possible  to  find  a  reasonable  practical  solution  that,  although  not 
optimal,  will  give  a  reasonably  good  result. 


II.  RECTIFICATION/REGISTRATION  OVERVIEW 


An  important  image  processing  technique  is  the  rectification  of  imagery 
generated  by  various  sensors.  Rect i f ica t ion  methods  are  usually  categorized 
under  the  more  general  term  of  "preprocessing"  techniques.  All  preprocessing 
techniques  involve  modifying  an  image  to  make  it  more  like  an  "ideal"  image, 
which  is  the  first  step  of  visual  data  processing.  Its  objectives  are:  (a) 
reconstruction  of  the  ideal,  high  fidelity  image  from  the  low-quality,  dis¬ 
torted  input  image;  and  (b)  improvement  or  enhancement  of  the  quality  of  the 
input  image  by  suppressing  noise  and  emphasizing  selected  features  to  facili¬ 
tate  later  stages  of  image  processing.  Preprocessing  involves  three  kinds  of 
modification:  rectification,  gray-level  modification,  and  sharpening/ 

smoothing  operations.  Gray-level  (or  intensity)  correction  compensates  for 
the  nonuniformity  of  sensor  sensitivity  and  contrast.  Sharpening  and 
smoothing  are  important  for  edge  detection  and  other  types  of  feature  extrac¬ 
tion  . 

In  this  section,  the  process  of  rectification  as  a  whole  is  reviewed,  the 
sources  of  image  distortion  are  examined,  and  formation  of  useful  data  bases 
for  specific  map-matching  problems  is  considered. 

A .  CATEGORIES  OF  ERRORS 

Numerous  error  sources  contribute  in  the  existing  distortions  in  an 
unprocessed  sensed  image  when  compared  to  a  preprocessed  reference  map  of  the 
same  region.  These  error  sources  are  grouped  into  four  convenient  categories 
according  to  Conrow  and  Ratkovic  [l]  as  depicted  in  Table  1.  These  categories 
are  global,  regional,  local,  and  nonst ructured  errors. 

Global  errors  affect  intensity  levels  of  all  elements  of  the  scene  uni¬ 
formly  and  can  generally  be  removed  by  the  rectification  process.  These 
errors  include  geometrical  and  radiometrical  (gain/bias)  distortions. 

The  second  category,  called  regional  errors,  uniformly  affects  the 
intensity  levels  within  homogeneous  regions  only  and  not  the  whole  scene  as 
with  global  errors.  An  example  of  this  category  is  "contrast  reversals"  which 
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affect  the  sensed  image  such  that  the  relative  order  of  intensities  between 
the  two  images  to  be  compared  is  not  preserved.  This  can  occur  in  infrared 
images  acquired  at  different  times  of  the  day  or  in  cases  where  a  SAR  image  is 
compared  with  a  visible  image  of  the  same  scene.  A  significant  variation  in 
contrast  may  lead  to  grossly  incorrect  matches.  There  are  some  possibilities 
of  error  removal  in  this  case  through  edge-matching  algorithms  and  cluster- 
reward  techniques  that  are  described  later. 

Local  errors  are  the  third  category  of  errors.  Local  errors  affect  each 
pixel  within  a  small  region  independently.  Additive  noise  and  speckle  in  SAR 
imagery  come  under  this  category.  The  spatial  size  of  the  speckle  is 
generally  on  the  order  of  the  limiting  resolution  of  the  coherent  SAR  imaging 
system,  and  this  signal-dependent  coherent-noise  makes  rectification  and 
restoration  of  SAR  images  more  difficult.  Some  algorithms  are  available  for 
the  partial  removal  of  speckle  available,  and  these  are  discussed  in 
section  VII. 

The  last  grouping  of  errors,  called  nonstructured  errors,  covers  all 
other  error  sources  not  already  included  in  the  global,  regional,  or  local 
categories.  An  example  is  a  cloud  over  portions  of  the  region  of  interest. 
However,  some  rectification  algorithms  (correlation  type  or  hybrid  type)  can 
still  perform  even  under  the  condition  that  part  of  the  scene  in  the  sensed 
image  is  missing  (under  the  cloud). 

B .  TYPES  OF  DISTORTIONS  REMOVED  BY  RECTIFICATION 

A  sequence  of  images  of  the  same  scene  taken  at  different  times  by  the 
same  sensor  will  contain  relative  distortions  because  of  differences  in  the 
viewing  and  scanning  conditions  at  the  sensor.  These  distortions  are  in  addi¬ 
tion  to  other  variations  and  error  sources  as  discussed  in  the  previous 
section.  Generally,  radiometric  and  geometric  distortions  (which  are  in  the 
category  of  global  errors)  can  be  accommodated  by  rectification  and  pre¬ 
processing,  whereas  all  other  types  of  errors  can  be  removed  somewhat  through 
the  right  choice  of  algorithm  [1].  This  section  discusses  details  of  radio- 
metric  and  geometric  distortions. 
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1.  RADIOMETRIC  DISTORTIONS 


These  distortions  are  caused  by  atmospheric-  and  sensor-induced 
filtering,  sensor  imperfections ,  camera  or  scanner  shading  effects  (nonuniform 
responses),  detector  gain  variations,  and  sensor  detection  gain  errors. 
Radiometric  distortions  appear  in  the  form  of  blemishes,  horizontal  stripes, 
shading,  and  nonuniform  intensity  distributions.  Radiometric  intensity 
corrections  for  nonuniform  sensor  sensitivity  can  be  implemented  in  real-time 
using  simple  table  lookup  techniques  [1-3].  Bias  and  gain  error  compensations 
can  also  be  implemented  in  the  same  manner.  The  effects  of  some  of  these  dis¬ 
tortions  on  the  image  are  shown  in  Figure  2. 

Radiometric  distortions  are  generally  predictable  and  can  be  corrected 
relatively  easily.  Radiometric  corrections  are  described  in  detail  primarily 
in  the  remote  sensing  literature  [ 1 , 2 J .  The  following  sections  of  the  report 
deal  mainly  with  geometric  corrections  of  data,  including  the  analysis  of  new 
techniques  available  and  their  error  removal  capabilities.  The  radiometric 
errors  are  considered  predictable  and  correctable. 

2.  GEOMETRIC  DISTORTIONS 

The  primary  causes  of  geometric  distortion  are: 

©  Sensor  related  distortion 

o  Alignment  variation 

o  Attitude  variation  of  the  spacecraft 

«  Ephemeris  variation 

Image  distortion  will  result  In  a  corresponding  registration  error  if  the 
distortion  is  not  estimated  and  removed.  The  effects  of  some  of  these  distor¬ 
tions  are  shown  in  Figure  2. 

a.  Sensor  Related.  Variation  in  the  motion  of  the  sensor  over  succ.es 
sive  passes  introduces  distortions.  For  example,  stretching  or 
compression  of  pixel  spacing  within  a  scan  line  results  from  varia¬ 
tions  in  tlie  velocity  of  the  scanner.  Another  example  of  distortion 
is  the  spacing  between  scan  lines  which  can  change  from  one  image  to 
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Characteristics  of  Image  Distortions 


the  other  as  a  result  of  variations  in  scan  period.  The  angular 
velocity  of  the  scan  also  produces  distortions  in  the  form  of 
irregularly  spaced  pixels  in  the  output  image. 

b.  Alignment  Variation.  The  variation  in  alignment  of  the  sensor  with 
respect  to  the  spacecraft  coordinates  axes  results  in  both  long  term 
drift  and  short  term  variations. 

c.  Attitude  Variation.  Variation  in  the  spacecraft  attitude  (yaw,  roll, 
and  pitch)  with  respect  to  a  previous  pass  will  cause  registration 
error  . 

d.  Ephemeris  Variation.  This  results  from  variations  in  the  location  of 
the  platform  with  respect  to  the  ground  with  successive  passes  over  a 
given  region.  This  error  must  be  corrected  for  the  rectification  and 
comparison  of  different  images. 

C.  RECTIFICATION/REGISTRATION  PROCESS 

The  general  process  of  rectification  for  a  multisensor  environment  is 
shown  in  Figure  3. 

The  first  step  of  the  process  is  to  match  a  sensed  image  to  a  reference 
image.  The  reference  image  is  typically  larger  than  the  sensed  image,  and  the 
sensed  image  may  have  magnification  or  rotation  differences  or  may  be  geomet¬ 
rically  distorted  compared  to  the  reference.  If  a  priori  geometrical  distor¬ 
tion  information  is  available  (from  calibration  or  inflight  data,  for 
example),  it  can  be  used  to  greatly  simplify  the  matching  process.  In  effect, 
any  a  priori  knowledge  reduces  the  number  of  degrees  of  freedom  in  the  search. 

Following  corrections  from  a  priori  information,  or  directly  starting 
with  the  original  sensed  image,  the  matching  process  begins  by  comparing  por¬ 
tions  of  the  sensed  image  to  the  reference.  Typically,  recognizable  features 
in  the  sensed  image  such  as  lines,  vertices,  edges,  or  shapes  are  compared  to 
tlie  reference  to  find  a  set  of  "match  point"  pairs  that  can  be  identified  in 

both  images.  By  matching  only  portions  of  the  sensed  image  to  the  reference, 
computations  are  reduced  considerably,  and  the  matching  is  made  resistant  to 
minor  differences  in  magnification,  rotation,  and  distortion. 

Image  matching  algorithms  (correlation,  symbolic  matching,  or  hybrid)  are 
used  to  compute  the  location  of  an  observed  match-point  (i.e.,  control  point 
or  tie  point)  on  the  sensed  image  from  the  actual  match  points.  The  actual 


Figure  3.  Rec t i f icat i on  Process 


match  points  are  physical  features  detectable  in  the  reference  map,  whose 
elevations  and  locations  are  known  precisely. 

As  shown  in  Figure  3,  after  the  "match-point"  pairs  are  found,  a  mapping 
operation  is  performed.  In  this  step,  several  match  points,  which  are  dis¬ 
tributed  on  an  irregularly  spaced  mesh  of  points,  are  used  to  calculate  a 
mapping  function,  relating  points  on  the  sensed  image  to  corresponding  points 
on  the  reference  map.  This  is  achieved  through  a  set  of  distortion  functions 
that  are  generally  piecewise  or  glohal  polynomials,  generally  having  terms  up 
to  and  including  cubic  powers  of  coordinate  variables.  More  details  on  the 
distortion  functions  are  given  in  section  ITT. 

After  match  point  pairs  and  distortion  functions  are  found,  the  next  step 
is  to  perform  a  resampling  operation  on  the  sensed  image  pixels  to  register  it 
with  the  reference  image.  An  interpolation  in  spatial  coordinates  is  per¬ 
formed  on  the  irregular  grid  of  sensed  image  match  points  to  determine  the 
location  of  pixels  that  are  geometrically  transformed  to  a  rectified 
coordinate  system.  These  pixel  locations  will  generally  never  fall  exactly  on 
a  pixel  location  in  the  sensed  image,  so  interpolation  or  resampling  is  per¬ 
formed  to  obtain  an  output  intensity  value  for  the  corrected  pixel.  Several 
methods  of  interpolation  with  different  amounts  of  smoothing  are  available. 

Section  V  of  this  report  is  an -overview  of  the  resampling  process  as  it 
relates  to  registration  and  rectification  of  sensed  imagery.  Various  one-  and 
two-dimensional  interpolation/resampling  techniques  are  reviewed  and  compared 
in  detail. 

D.  DATA  BASE  FORMATION 

Wide  variations  in  temporal  and  spatial  characteristics  of  a  scene  or 
region  of  the  terrain  make  it  very  difficult  to  develop  a  complete  data  base 
toward  a  fully  automated  rectification  system.  Nevertheless,  some  useful  data 
bases  for  specific  map-matching  problems  are  available  [1-7], 

Figure  4  shows  a  possible  reference  data  base  (i.e.,  a  reference  map) 
which  includes  operator  annotations,  vector /graph i cs ,  and  gray-level  image 
data.  Figure  5  shows  a  more  general  data  base  including  types  of  target, 
temporal  signature  variation,  sensor  wavelengths,  and  material  interfaces. 


OPERATOR  ANNOTATIONS 

•  VECTOR  DATA 

•  RASTER  DATA 


VECTOR/RASTER  GRAPHICS 


•  SATELLITE  DATA 

•  AIRCRAFT  DATA 

Figure  4.  Reference  Data  Base 
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Figure  5.  Data  Base  Formation 


Hr.  REAL-TIME  GEOMETRIC  CORRECTION 


In  this  section,  geometrical  distortion  in  imaging  systems  due  to  optical 
and  orientation  effects  is  described,  and  design  concepts  for  near-real -time 
correction  are  presented.  Correction  for  matching  display  and  superposition 
of  multiple  images  are  discussed.  General  computing  and  system  hardware 
requirements  are  also  described. 

A.  GEOMETRICAL  DISTORTIONS 

In  all  imaging  systems,  a  square  coordinate  grid  placed  in  front  of  the 
sensor  does  not  appear  square  on  display.  The  resulting  effect  is  called  geo¬ 
metrical  nonlinearity  or  geometrical  coordinate  distortion.  A  simulation  of 
the  effect  can  he  achieved  by  plotting  a  square  grid  on  a  flexible  surface 
(say  a  rubber  sheet)  and  then  stretching  it  to  destroy  the  rectilinear 
pattern.  Geometrical  distortion  usually  affects  only  the  position  and  not  the 
intensity  of  pixels,  although  in  scanning  systems  the  two  effects  are  inter¬ 
related  because  the  density  of  scan  lines  in  a  given  region  influences  the 
apparent  brightness.  Regardless  of  whether  a  human  observer  or  a  measurement 
system  ultimately  uses  the  image  information,  some  correction  of  these  non- 
linearities  is  generally  necessary.  If  a  number  of  different  sensors  view 
the  same  general  scene  and  are  to  be  superimposed  for  viewing  (as  in  multi- 
spectral  imaging)  it  is  important  that  the  effects  of  different  relative 
distortions  in  all  systems  be  corrected.  Typical  sources  of  geometrical  dis¬ 
tortions  in  imaging  systems  include:  (a)  nonl i nea r i t ies  in  the  sensor  and 
scanning  electronics;  (h)  aberrations  in  the  imaging  optics,  such  as  pin¬ 
cushion  and  barrel  distortion;  and  (c)  differences  in  the  relative  orientation 
between  multiple  sensors. 

The  objective  of  this  section  is  to  develop  procedures  that  correct 
distorted  images  for  display  in  real  time.  When  the  output  of  many  sensors 
are  to  be  superimposed,  the  correction  must  be  independently  accomplished  for 
all  prior  to  display,  and  it  is  assumed  that  similar  procedures  will  be 
applied  to  each  sensor.  The  distortion  is  modeled  as  memorvless  mapping  of 
object  scene  points  to  display  pixels,  with  no  inherent  smearing  or  averaging. 
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The  distortion  is  fixed  in  time,  or  at  least  slowly  varying,  so  that 
calibration  of  the  correction  system  is  not  required  during  use. 

Assume  that  the  geometrically  ideal  image  intensity  is  given  by  f(x,y), 
where  (x,y)  are  spatial  variables  in  a  rectilinear  coordinate  system  defined 
over  the  entire  display  field.  The  sensor  measures  this  ideal  image  inten¬ 
sity,  where  it  becomes  the  distorted  image  intensity  g(u,v)  at  position  (u,v) 
in  the  distorted  coordinate  system  over  the  sensor  field  of  view.  The  dis¬ 
tortion  relation  between  systems  is  given  parametrically  by 

x  =  ct(u,v)  (la) 

y  =  0(u, v)  (lb) 

Substitution  of  Eq .  (1)  in  f(x,y)  ar.d  equating  the  intensity  measurement  gives 
the  relation 

f (x , y)  =  f[a(u,v),  B(u,v)]  =  g(u,v)  (2) 

between  f  and  g.  The  distortion  given  by  Eq .  (1)  may  be  nonlinear  functions 
of  (u,v)  but  must  be  one-to-one  mappings  of  points  from  one  coordinate  system 
to  another  as  depicted  in  Figure  6.  If  the  distortion  is  a  linear  function  of 
the  coordinates,  then  straight  lines  are  transformed  to  straight  lines  and  the 
transformation  reduces  to  a  combination  of  magnification,  rotation,  and  trans¬ 
lation.  With  these  given  properties  of  Eq.  (1),  the  system  may  be  inverted 
and  solved  to  produce 
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Figure  6.  Sensor  Distortion 
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Figure  7.  Ideal  Correction 
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which  is  the  ideal  inverse  distortion.  The  optimum  distortion  correction 
system  simply  takes  g(u,v)  from  the  sensor  and  substitutes  in  Eq.  (3)  to  give 

g(u,v)  =  g[<j>(x,y),  >Kx , y ) ]  =  f(x,y)  (4) 

where  f(x,y)  is  the  corrected  estimate.  This  operation  is  shown  in  Figure  7 
and  is  the  goal  of  all  the  correction  systems  proposed  here. 

The  geometrical  correction  to  recover  f(x,y)  from  g(u,v)  is  accomplished 
by  first  constructing  the  distortion  model  of  Eq.  (3)  and  then  placing  the 
pixel  value  g(u,v)  in  the  correct  position  (x,y)  to  give  the  estimate  of  the 
ideal  image  f(x,y).  The  second  process  is  called  "resampling." 

In  a  scanned  display  system,  the  spacing  of  the  scan  lines  and  scanning 
speed  influences  the  brightness  observed  by  the  viewer.  In  this  case,  Eq.  (2) 
describing  the  degradation  must  be  modified  from  a  simple  point  movement  by 
adding  a  multiplicative  term.  The  requirement  is  that  the  power  collected  by 
the  imaging  system  which  is  radiated  from  a  particular  spatial  region  of  the 
object  must  be  conserved  at  any  time  instant.  Thus,  the  power  radiated  by  a 
small  region  dxdy  around  object  point  (x,y)  is  measured  as 

f(x,y)  dxdy  =  g(u,v)  dudv  (5) 


in  the  distorted  coordinates.  Using  the  distortion  relations  Eq.  (1),  the 
elemental  areas  dudv  and  dxdy  are  related  by 

dxdy  =  J(u,v)  dudv  (6) 


where 


3a 

3a 

■3u 
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3u 

3v 

is  a  Jacobian  determinant. 


Substituting  into  Eq .  (7)  gives 


g(u,v)  =  J(u,v)  f  [  <*(  u,  v)  ,  3(u,v)J  (8) 

as  the  distorted  image.  This  expression  differs  from  Eq.  (4)  found  previously 
by  the  multiplicative  factor  J(u,v).  The  effect  of  J(u,v)  is  a  multiplicative 
gain  which  varies  with  position,  and  it  may  be  included  with  other  Intensity 
nonlinearities  of  the  system. 

Figure  8  is  a  block  diagram  of  a  distortion  correction  and  resampling 
system.  In  the  next  few  sections  we  describe  major  subsystems. 

B.  CONSTRUCTION  OF  DISTORTION  MODEL 

To  find  the  inverse  distortion  model  of  Eq.  (3),  pairs  of  control  points 
or  landmarks  in  both  the  reference  and  sensor  coodinate  systems  must  be  found. 

Figure  9  shows  labeled  pairs  of  points  (A-A'),  (B-B'),  etc.,  in  the 
sensor  and  corrected  sensor  coordinates.  We  assume  that  these  match  points 
are  available  from  correlation,  sequential  similarity  detection,  symbolic 
matching,  or  hybrid  algorithms  that  are  described  later  in  this  report.  From 
these  data,  a  model  of  the  distortion  can  be  obtained. 

An  analytical  model  of  the  distortion  correction  can  be  obtained  from 
mapping  polynomials  in  (x,y)  of  degree  N  having  the  form 

N  N-P 

u  =  <)>(x  ,y )  =  I  l  a  xPyq  (9a) 

p=0  q  =0  pq 

N  N-P 

v  =  iKx,y)  =  l  l  b  xPyq  (9b) 

p=0  q=0 

where  and  h  are  the  constant  polynomial  coefficients  [4],  [6],  [7], 

Special  cases  of  Eq.  (9)  include  the  linear  or  affine  model 
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Figure  8.  System  Block  Diagram 


V  =  b10x  +  b01y  +  b00 


(10b) 


and  the  bilinear  distortion  model 

u  =  auxy  +  a10x  +  a(),  y  +  a()Q  (11a) 

v  =  bnxy  +  bJ0x  +  b0]y  +  bQ0  (lib) 

Many  local  distortions  in  an  image  are  accurately  modeled  by  linear  or 
bilinear  equations.  For  many  practical  cases  involving  the  human  observer  in 
real  time  or  when  the  distortion  is  not  severe,  a  second  degree  (N=2)  approxi¬ 
mation  is  probably  sufficient.  Lillestrand  [7]  describes  the  error  as  a 
function  of  increasing  polynomial  terms  and  shows  that  little  is  gained  by  N 
greater  than  2  or  3 .  However,  for  large  areas  and  more  severe  distortions, 
higher  order  polynomials  are  necessary.  Alternatively,  a  large  image  may  be 
divided  into  a  number  of  smaller  patches  to  which  a  linear  or  bilinear 
equation  is  applied.  Figure  10  shows  how  individual  pixel  locations  are  found 
by  bilinear  interpolation  on  a  mesh  of  points  obtained  from  mapping 
polynomials. 

Finding  the  coefficients  in  Eq .  (9)  involves  fitting  two  dimensional 
functions  to  a  set  of  measured  values.  Various  estimation  techniques  can  be 
used  [8],  [9],  most  of  which  provide  a  minimum  mean-squared  rror  (MMSE)  fit. 

Two  approaches  to  the  fitting  are  described  here. 

Direct  modeling  is  the  first  technique.  It  requires  prior  knowledge 
about  the  position  of  control  points  in  the  sensed  and  reference  images.  An 
alternative  is  indirec'.  modeling,  in  which  a  structural  model  of  distortion  is 
derived  from  knowledge  of  the  imaging  process.  For  example,  it  is  possible  to 
infer  a  distortion  model  from  knowledge  of  the  angle  of  elevation  and  other 
parameters.  The  parameters  in  the  model  may  be  determined  by  logging  data 
from  the  imaging  device,  for  example,  the  camera  parameters  (focal  length)  and 
the  vehicle  parameters  (x,y,  and  7.  coordinates  in  three-dimensional  space,  and 
roll,  pitch,  and  yaw).  This  approach  is  common  in  interpreting  aerial 
photographs. 
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MMSE  fitting  expresses  the  4>(x,y)  and  <J>(x,y)  surfaces  by  polynomials 
whose  squared  distance  from  the  true  surface  is  a  minimum.  To  obtain  the 
coefficients,  control  point  pairs  are  identified  in  sensed  and  reference 
images.  R  control  point  pairs  are  selected,  and  their  values  of  u  =  <j>(x,y) 
and  v  =  ip(x,y)  are  recorded.  For  a  second  degree  (N=2)  approximation  to  this 
distortion,  there  are  six  coefficients  to  determine,  and  the  measured  data  for 
(u,v)  are  arranged  in  the  form 


lx'  y ' 


2  2 

*  y  xy 


Equation  (12)  can  be  expressed  as  the  vector  equation 


u  =  Da 


where  u  is  an  R-vector  of  measured  output  points  (R  >  6),  D  is  the  matrix  of 
input  calibration  points,  and  _a  is  the  coefficient  vector.  Using  MMSE  estima¬ 
tion  theory,  the  best  estimate  for  a  is 


a  =  (DTD)  1  DTu 


where  T  and  -1  denote  transpose  and  inverse,  respectively.  The  data  for  v^  are 
taken  at  the  same  time  and  used  to  find  _b  for  the  polynomial  of  Eq.  (9).  If 
greater  accuracy  is  required  for  measurement  purposes  and  N  =  2  is  unsatisfac¬ 
tory,  more  terms  in  the  expansion  can  be  used  and  the  size  of  D  enlarged.  For 


multiple  sensors  which  are  superimposed,  the  entire  procedure  is  repeated  for 
each.  It  is  also  possible  to  improve  the  approximation  over  the  polynomials 
of  Eq.  (9)  with  the  same  number  of  terms  in  many  physical  systems  [9], 

C.  DISTORTION  CORRECTION  SUBSYSTEMS 

The  subsystems  labeled  4  and  5  in  Figure  8  perform  the  actual  correction 
based  on  the  pixel  correction  functions  derived  in  the  previous  section.  The 
system  consists  of  sensed  data  buffers  analyzed  in  a  speed  hierarchy  as  shown. 
The  low-speed  data  buffer  may  be  a  disc,  tape,  or  mass  storage  system  that 
transfers  data  to  a  higher  speed  but  lower  capacity  system.  The  high-speed 
data  buffer  is  generally  random  access  memory  (RAM)  in  a  computer. 

The  pixel  movement / inte rpolat ion  system  performs  a  correction  by  stepping 
through  the  discrete  output  (x,y)  locations  and  finding  the  corresponding  dis¬ 
torted  locations  (u,v).  The  polynomial  mappings  can  either  be  precomputed  and 
stored  in  a  separate  lookup  table,  or  they  can  be  computed  on  the  fly  from  the 
polynomials. 

When  the  distorted  location  (u,v)  is  found,  the  intensity  value  and  local 
neighbors  stored  there  are  used  to  compute  a  geometrically  corrected  value 
that  is  transferred  to  the  output  buffer.  The  general  process  is  called 
resampling  and  is  discussed  in  detail  in  Section  V.  The  simplest  form  of 
resampling  is  to  do  no  interpolation  -  the  nearest  neighbor  pixel  value  to  the 
location  specified  by  the  polynomials  is  simply  moved  to  the  output  buffer. 
Other  methods  of  interpolation  are  given  in  Section  V. 

In  general,  the  sensed  data  buffers  need  to  store  only  as  many  pixels  as 
needed  to  fill  one  line  of  the  corrected  data  buffer.  Other  important  con¬ 
siderations  in  hardware  implementation  of  the  distortion  are  the  number  of 
image  pixels,  speed  of  implementation,  and  the  maximum  amount  of  gross 
rotation  distortion  between  sensed  and  reference  images.  In  general,  the 
sensed  image  buffer  should  be  as  large  as  possible  to  reduce  accesses  to 
slower-speed  mass  storage  devices  such  as  discs.  Large  sensed  images  may  not 
entirely  fit  in  a  high  speed  buffer,  so  that  many  disc  accesses  may  be  needed 
for  large  distortions.  A  90-deg  rotation  of  sensed  image  is  the  worst 
possible  case  because  it  effectively  requires  a  transpose  of  a  large  array,  or 


equivalently,  each  corrected  pixel  comes  from  a  different  line  of  the  sensed 
image.  For  small  distortions,  the  high  speed  buffer  can  be  much  smaller 
because  fewer  lines  of  the  sensed  image  are  needed  to  generate  one  corrected 
line.  Sophisticated  data  base  organization,  parallel  processors,  and  parallel 
disc  access  hardware  may  be  effectively  used  in  speeding  up  the  correction 


process . 


IV.  RECTIFICATION  AND  CHANGE  DETECTION  ALGORITHMS 


Three  major  categories  of  rectification  algorithms  have  emerged  from  the 
study. 

•  Signal-processing-based  (SP) 

•  Artificial-intelligence-based  (AI) 

•  Hybrid  techniques 

The  signal-processing-based  algorithms  use  various  correlation  methods  to 
find  the  best  match  between  reference  map  and  sensed  image.  The  Al-based 
algorithms  compare  symbolic  representations  of  features  extracted  from  both 
images.  Hybrid  techniques  combine  approaches  in  an  effort  to  make  the  best 
use  of  both. 

In  this  section,  various  image  matching  algorithms  for  multisensor 
environments  are  studied  and  analyzed  in  detail.  Change  detection  algorithms 
are  also  reviewed. 

A.  TYPES  OF  ALGORITHMS 

Two  basic  types  of  matching  procedures  exist:  signal  processing  or 
correlation-based  techniques;  and  artificial  intelligence  or  symbolic-based 
techniques.  More  recently,  hybrid  techniques  have  been  developed  that  attempt 
to  make  the  best  use  of  both  procedures  (see  Figure  11).  The  signal  process¬ 
ing  techniques  work  by  correlating  intensity  values  of  original  or  enhanced 
portions  of  the  sensed  Images  against  the  reference.  The  best  match  based  on 
a  match  quality  measure  is  then  recorded.  There  are  a  number  of  ways  to 
perform  the  correlation,  and  a  fast  algorithm  known  as  sequential  similarity 
detection  (SSDA)  offers  a  speed  improvement  by  a  factor  of  IDO  over  conven¬ 
tional  correlation.  The  artificial  intelligence  tech,  iques  compare  symbolic 
representat ions  of  features  extracted  from  both  reference  and  sensed  images. 
These  features  can  be  edges,  lines,  vertices  of  line  intersections,  shapes, 
etc.  Given  a  distance  function  or  other  >lative  match  quality  measure,  graph 
matching  or  searching  techniques  such  relaxation  labeling  are  used  to 
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compare  reference  and  sensed  symbolic  data  sets.  The  symbolic  data  have  a 
much  lower  dimensionality  than  pixel-based  signal  processing  data,  thus  com¬ 
putations  should  be  reduced  for  symbolic  matching.  One  additional  tradeoff 
between  the  techniques  concerns  noise  resistance.  Correlation  techniques 
provide  inherent  noise  suppression  and  can  find  good  matches  with  significant 
amounts  of  image  noise  if  the  relative  magnification,  rotation,  and  distortion 
differences  are  not  great.  Symbolic  matching  strongly  relies  on  good,  noise- 
free  feature  extraction.  With  noisy  edges,  lines,  vertices,  etc.,  symbolic 
techniques  will  fail,  thus  good  preprocessing  for  noise  suppression  and 
robust,  noise-resistant  feature  extract  is  essential  for  symbolic  matching. 
Hybrid  techniques  combine  both  approaches  to  njake  the  best  use  of  both. 

Tables  2  and  3  summarize  this  result  for  various  types  of  sensors  under  study 
and  for  the  different  error  categories  discussed  in  section  II. 

B.  SIGNAL-PROCESSING-BASED  ALGORITHMS 

The  standard  signal  processing  technique  for  registering  images  is  to 
form  a  correlation  measure  between  them  as  a  function  of  translational  shifts 
and  find  the  location  that  gives  the  maximum  correlation  value.  Several 
references  discuss  this  problem  [10,12],  and  the  book  by  Pratt  [13]  summarizes 
the  results. 

The  simplest  form  of  a  two-dimensional  correlation  measure  is  given  by 
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where  Fj  (i,k)  and  F 2  ( i , k )  are  two  discrete  images  to  be  registered,  and 
(j ,k)  are  indices  in  a  J  *  K  pixel  window  area  W  that  is  located  within  an 
M  x  N  point  search  area  S.  Figure  12  shows  the  search  and  window  areas.  In 
general,  the  correlation  function  is  computed  for  all  (M-J+l)  (N-K+l)  possible 


Table  2.  Sensor-Dependent  Rectification  Methods 


SENSOR  TYPE  LOW  SNR  MODERATE  SNR  HIGH  SNR 


OPTICAL/NEAR  IR 

P,A 

CORRELATION 

HYBRID 

FEATURE  MATCHING 

P 

HYBRID 

HYBRID 

FEATURE  MATCHING 

MIDDLE  IR 

A 

CORRELATION 

HYBRID 

FEATURE  MATCHING 

P 

HYBRID 

HYBRID 

FEATURE  MATCHING 

THERMAL  IR 

A 

CORRELATION 

HYBRID 

FEATURE  MATCHING 

MICROWAVE 

P 

CORRELATION 

HYBRID 

FEATURE  MATCHING 

SAR 

A 

1-D  CORRELATION 

2-D  HYBRID 

HYBRID 

(not  adequate  SNRI 

P  =  PASSIVE  SENSORS 
A  =  ACTIVE  SENSORS 
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Table  3.  Algorithm  Error  Analysis 


ERROR  CATEGORY 

ALGORITHM 

GLOBAL 

(Geometric, 

radiometric) 

REGIONAL 

(Contrast 

reversal) 

LOCAL 

(Noise) 

NONSTRUCTURED 

lOthers) 

CORRELATION 

MEDIUM 

POOR 

GOOD 

GOOD 

FEATURE  MATCHING 

MEDIUM 

GOOD 

POOR 

POOR 

HYBRID 

GOOD 

GOOD 

MEDIUM 

GOOD 

translations  of  the  window  area  within  the  search  area  to  find  its  maximum 
value  and  the  displacement  that  gives  the  best  match. 

One  possible  problem  with  the  correlation  technique  is  that  the  correla¬ 
tion  function  may  be  very  broad,  making  detection  of  the  peak  difficult.  This 
will  particularly  happen  if  the  search  and  window  images  contain  large  uniform 
regions  with  very  few  details.  Another  problem  is  that  differences  in  scale 
size,  geometric  distortion,  rotation,  and  relative  intensity  between  the  two 
images  will  make  registration  difficult  and  effectively  suppress  the  correla¬ 
tion  peak.  Pratt  [13]  has  described  some  techniques  for  avoiding  the  first 
problem.  They  consist  of  prefiltering  the  window  and  search  areas  with  a 
Laplacian  spatial  high-pass  fiiter  that  effectively  performs  edge  enhancement. 

Another  difficulty  with  the  correlation  method  of  image  registration  is 
the  large  amount  of  computation  needed.  In  Eq .  (15)  the  correlation  measure 
Rs  (m,n)  must  be  computed  for  all  (m,n)  before  the  maximum  can  be  selected, 
and  the  amount  of  computing  is  the  same  for  all  degrees  of  misregistration. 

To  overcome  some  of  these  problems,  a  class  of  algorithms  known  as 
sequential  similarity  detection  (SSD)  has  been  developed  by  Barnea  and 
Silverman  [14],  These  algorithms  provide  a  test  for  misregistration  with 
fewer  computations. 

The  basic  form  of  the  algorithm  is  to  compute  an  error  measure 

e  (m,n)  *  l  [  |  F  j  ( i  ,k )  -  F2  (j  ~m ,  k-n)  |  (16) 

j  k 

where  the  sums  are  taken  over  the  window  area  and  Fj  and  F2  are  the  window  and 
search  functions,  respectively,  as  before.  However,  the  sum  is  not  taken  over 
a  regular  scan  indexing  of  (j,k).  Instead,  the  absoljte  value  of  the  differ¬ 
ence  in  Eq.  (16)  is  accumulated  for  randomly  selected  points  In  the  window 
area.  If  the  accumulated  error  exceeds  a  fixed  threshold  before  all  J  *  K 
points  in  the  window  area  are  examined,  then  it  is  assumed  that  a  particular 


shift  (m,n)  is  a  poor  candidate  for  matching,  and  another  shift  of  the  window 
is  checked.  If  the  error  grows  slowly,  then  the  number  of  points  examined 
when  the  threshold  is  finally  exceeded  is  recorded  and  used  as  a  rating  of  a 
particular  shift  (m,n).  When  all  possible  shifts  have  been  examined,  the 
window  with  the  largest  rating  is  chosen  as  best.  The  highest  possible  rating 
is  equal  to  the  number  of  window  points  J  *  K.  In  this  way,  shifts  that  are 
grossly  incorrect  are  discarded  rapidly  after  only  a  few  computations. 

The  SSD  algorithm  can  also  be  combined  in  a  hybrid  approach  as  a  pre¬ 
processor  for  correlation  registration.  The  SSD  algorithm  can  be  used  to 
discard  gross  misregistration,  while  a  correlation  algorithm  can  be  used  for 
the  final  refinement.  A  reduction  in  computations  by  a  factor  of  100  over 
correlation  has  been  experimentally  observed  using  SSD  techniques. 

C.  ARTIFICIAL-INTFl.hlGENCK-BASED  ALGORITHMS 

As  outlined  earlier,  feature  matching  algorithms  do  not  directly  use 
intensity  data.  Instead,  they  use  scene  features  such  as  edges,  boundaries, 
and  line  vertices.  The  first  stage  of  processing  generally  extracts  these 
features.  Then  distance  functions  between  sensed  and  reference  image  features 
are  defined  and  a  search  for  the  best  match  is  made.  The  search  techniques 
can  include  graph  matching,  and  nonlinear  or  linear  optimization.  Some 
examples  of  scene  matching  techniques  are  discussed  in  detail  here. 

There  are  two  major  types  of  artificial  intelligence  (Al)-based 
algorithms  for  image  matching  that  require  little  or  no  a  priori  control  point 
information.  These  techniques  are:  clustering  features  and  feature  groups; 
and  st rue tura 1 / symbol ic  matching. 

The  clustering  approach  begins  with  features  extracted  from  both 
reference  and  sensed  images.  These  features  can  be  intensity  itself,  edges, 
texture,  etc.  No  labeled  control  points  are  present  in  the  two  images.  The 
general  algorithm  provides  a  reward  for  region  correspondence  based  on  the 
clustering  of  the  joint  histogram  of  features  in  the  two  images.  The 
algorithm  tries  to  measure  the  cor respondence  of  features  of  each  type  in  the 
two  images  and  finds  the  rotation,  translation,  and  scaling  needed  to  create  a 

Uk 


correspondence.  It  increments  the  cluster  model  by  one  unit  at  each  such 
angle,  x-y  coordinate,  or  scaling  parameter,  and  groups  values  corresponding 
to  the  identification  of  two  features  with  one  another.  This  approach  can 
immediately  follow  front-end  feature  extraction.  Basic  references  are 
[15-17].  Reference  [18]  discusses  sequential  feature  detection,  which  may  be 
useful  in  speeding  up  these  routines. 

Structural  and  symbolic  matching  is  to  date  a  research-level  algorithm 
process  involving  automatic  interpretation,  but  it  is  principally  applied  to 
partially  specified  scenes  [19]  and  structure  interpretation  from  image 
boundaries  [20].  A  data  structure  system  has  been  created  as  part  of  the 
ACRONYM  [21]  model-based  vision  system  involving  five  levels  of  graphs: 
"context,"  "object,"  "observability,"  "interpretation,"  and  "picture"  [19]. 
Also,  much  work  has  been  done  using  linear  features  [22]. 

Additional  detailed  work  on  scene  matching,  performed  by  Hughes  Research 
Laboratories  [23,24],  has  developed  scene  matching  systems  that  extract  line 
and  vertex  features  derived  from  the  scene.  Feature  weighting,  based  on 
matchpoint  location  and  model  feature  content,  is  used  to  define  geometrical 
transformations  between  reference  and  sensed  models  that  lead  to  highly  accu¬ 
rate  matchpoint  location.  An  important  part  of  this  work  is  the  use  of  data 
structures  for  regions,  lines,  vertices,  and  connected  groups  that  contain 
parameters  and  links  to  other  data  elements.  The  technique  is  experimentally 
highly  tolerant  to  differences  in  scene  contrast,  scale,  and  orientation 
differences.  More  details  of  these  algorithms  in  their  specific  application 
to  SAR  image  matching  are  given  in  section  VI. 

E>.  CHANGE  DETECTION  ALGORITHMS 

Once  the  sensed  an!  reference  Lmages  are  rectified  and  spatially 
registered,  both  signal  processing  and  symbolic  approaches  are  available  for 
change  detection,  in  a  manner  similar  to  that  used  to  obtain  pairs  of  match 
points  (see  Figure  13).  Any  change  detection  method  must  contend  with  differ¬ 
ences  in  illumination,  sensor,  wavelength,  and  surface  conditions  [6,7].  Some 
of  these  effects  can  be  compensated  for  by  a  priori  knowledge.  Signal 
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Figure  13.  Change  Detection  Between  Geometrically  Rectified  Scenes 


processing  approaches  for  registered  visible  and  IR  images  include  pixel 
differencing,  although  speckle  and  other  texture  artifacts  in  SAR  images  may 
cause  problems.  Symbolic  techniques  for  change  detection  involve  segmenting 
the  images  into  regions  or  shapes  with  similar  feature  properties,  and 
extracting  symbolic  descriptions  of  these  regions.  The  symbolic  descriptions 
may  be  edges,  lines,  shapes,  or  other  higher-level  visual  features.  These 
symbolic  descriptions  are  compared  by  graph  matching,  searching,  etc.,  similar 
to  the  general  techniques  described  for  finding  match  points.  This  type  of 
change  detection  allows  additional  knowledge,  such  as  contextual  relationships 
and  semantics,  to  be  included  in  the  change  detection  process.  Also,  hybrid 
approaches  that  combine  signal  processing  and  symbolic  processing  have  been 
studied.  More  investigation  into  techniques  of  change  detection  is  needed 


V.  RESAMPLING  PROCESS 


In  the  process  of  registration  and  rectification  of  remotely  sensed  data, 
a  problem  arises  as  a  result  of  nonuniform  spacing  of  the  input  pixel  values 
in  both  along-scan  and  across-scan  directions.  This  distortion  of  a  regular 
grid  and  pixel  points  is  a  result  of  implementation  of  geometric  and 
radiometric  corrections  on  the  image  data.  It  then  becomes  necessary  to 
evaluate  new  data  values  of  the  sample  in  order  to  achieve  a  regular  rectan¬ 
gular  grid  of  output  pixels  with  the  desired  number  of  lines  and  sample/lines 
for  the  purpose  of  registration.  This  process  is  called  "resampling." 

In  this  section,  an  overview  of  the  resampling  process  a”s  it  relates 
to  registration  and  rectification  of  onboard  sensor  imagery  is  discussed. 
Various  one-  and  two-dimensional  resampling  techniques  are  reviewed  and 
compared . 

A.  BACKGROUND 

Resampling  involves  constructing  an  ideal  image  by  determining  for  each 
pixel  (x,y)  the  corresponding  pixel  in  the  distorted  image.  The  intensity 
value  of  the  pixel  in  the  distorted  image  is  then  copied  into  its  undistorted 
position  in  the  ideal  image.  Unfortunately,  a  pixel  position  (i,j)  in  an 
ideal  image  will  usually  not  map  to  an  integer  coordinate  (m,n)  -  the  exact 
pixel  position  whose  value  is  sampled  In  the  distorted  image  -  but  to  a  point 
(u,v)  between  pixel  locations  (see  Figure  14).  How,  then,  is  the  value  at 
(u,v)  -  the  value  to  be  copies  into  pixel  (i,.i)  ~  to  be  determined?  There  are 
two  common  solutions  to  this  problem.  One  is  to  copy  the  value  of  the  nearest 
neighbor  to  (u,v)  into  the  ideal  image.  The  other  is  to  interpolate  a  value 
based  on  the  values  of  pixels  in  a  window  around  (u,v). 

During  the  past  decade,  a  great  deal  of  research  was  carried  out  regard¬ 
ing  the  application  of  digital  interpolation  techniques  to  remotely  sensed 
digital  images  [25-28]. 

Perfect  image  reconstruction  is  possible  with  a  number  of  interpolating 
kernels  (reconstruction  filters)  for  band-limited  image  samples  with  a  proper 
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sampling  period.  For  example,  a  sine  function  can  be  used  as  an  interpolating 
kernel  for  exact  reconstruction  of  an  infinite  number  of  sufficiently  close 
samples  of  a  band-limited  image  [35].  In  practice,  however,  because  of 
limitation  in  computation  time  and  storage,  it  is  often  difficult  or  imprac¬ 
tical  to  produce  exact  reconstruction  of  an  imaging  system  and,  instead, 
simple  approximating  techniques  are  used. 

B.  RESAMPLING  TECHNIQUES 

Three  approximating  techniques  are  commonly  used  in  resampling  of  one-  or 
two-dimensional  data:  nearest-neighbor,  bilinear  Interpolation,  and  cubic 
convolution  (spline  interpolation). 

Figure  15  shows  how  pixel  values  are  determined  for  a  location  (X)  in 
between  pixels  for  the  above  three  techniques.  In  this  figure,  pixels  are 
presented  by  dots,  and  lines  connecting  pixels  show  the  pixels  used  in  inter¬ 
polation  to  determine  an  image  data  value  at  the  location  presented  by  deltas. 
Lines  connecting  the  deltas  show  another  interpolation  process  required  to 
find  the  data  value  at  location  X. 

For  nearest  neighbor  resampling,  the  data  value  of  the  pixel  closest  to 
the  resampled  pixel  (location  X)  is  chosen  for  the  result  of  the  interpolation 
operation.  The  interpolation  kernel  is  linear  and  is  shown  in  Figure  16. 

This  kernel  produces  position  (horizontal  and  vertical)  errors  throughout  the 
operation,  as  much  as  *0.5  pixel  spacing,  which  results  in  rapidly  changing 
contouts  (jitters)  in  the  region  of  high  contrast  [25,26], 

The  bilinear  interpolation  kernel  shown  in  Figure  16  requires  a  bilinear 
combination  of  four  pixel  values  closest  to  the  desired  location  X 
[Figure  15(b)].  The  smoothing  effects  of  bilinear  interpolation  cause  some 
image  degradation  in  the  form  of  edge  smoothing  and  loss  of  maxima/minima 
structure  [26-28]. 

The  cubic  convolution  process,  which  is  an  approximation  of  the  sine 
function  by  cubic  polynomials  with  a  kernel  shown  in  Figure  16,  requires  the 
values  in  a  grid  of  4  x  4  pixels  about  the  desired  resample  pixel  location  X 
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[Figure  15(c)].  This  approach  [27]  in  the  earlier  reports  was  shown  to  give 
better  results,  even  in  comparison  with  sinx/x  truncated  at  10  pixels. 

C.  INTERPOLATION  PROCESS 

Once  the  match  points  for  the  distortion  are  computed,  it  is  necessary  to 
resample  the  irregularly  spaced  data  at  coordinates  (u,v)  so  as  to  produce 
output  image  data  at  coordinates  (x,y)  defined  on  a  regular  grid  of  pixels. 

The  resampled  image  data  are,  in  general,  computed  by  an  interpolation  proce¬ 
dure  . 


g(x,y)  =  I  f  (u,v)  K(x-u,  y-u) 
u,v 

where 

K  =  interpolation  kernel 
f  =  available  raw  data  values 
g  =  resampled  data  values 

Note  that  for  Integer  values  of  x,y  the  correspond! ng  image  coordinates  u  =  x 
+  <5x  and  v  =  y  +  <5y  are  noninteger  values,  where  5x,  <5y  are  image  distortion 
approximations . 

Although  several  two-dimensional  interpolation  kernels  for  regularly 
and  irregularly  spaced  data  are  available  [12,  32,33],  in  practice,  the  two- 
dimensional  kernel  K  is  considered  as  a  separable  kernel 

K(x-u,  y-u)  =  k(x-u)  h(y-u) 

and  the  resampling  process  is  implemented  as  two  one-dimensional  interpola¬ 
tions:  first  in  the  direction  of  raw  data  scan  Lines,  and  then  in  the 

orthogonal  direction. 

Note  that  the  ideal  interpolation  kernel  for  band-1 imi t i ng  data  is  of  the 


form 


V  „  sin  IT  (x-u)  sin  rr  (y-u) 

K(x-u,  y-u)  =  C  - 2 - - 

t  (x-u)  (y-u) 

However,  this  function  has  significant  magnitude  until  very  high  |x,u|  and 
| y— v |  and,  thus,  the  summation  terms  for  each  interpolation  tend  to  be  very 
large . 

In  one  dimension,  the  simplest  interpolating  kernel  is  the  pulse  func¬ 
tion,  which  results  in  nearest  neighbor  resampling.  A  triangle  function 
provides  a  bilinear  interpolating  kernel  (Figure  16).  The  cubic  convolution 
kernel  is  described  in  the  next  section;  its  smooth  curve  actually  coincides 
with  the  original  data  values. 

D.  CUBIC  CONVOLUTION  INTERPOLATION 

Cubic  convolution  interpolation  is  an  approximation  to  the  resampling 
process  by  fitting  a  piecewise  polynomial  to  the  sine  function  through  inter¬ 
polation  while  imposing  certain  constraints  [26].  The  contribution  of  a 
spline  kernel  to  the  approximation  method  is  twofold:  (a)  reduction  of  the 
summation  term  in  the  ideal  Interpolation  from  an  infinite  number  to  a  summa¬ 
tion  over  four  input  sample  pixels;  and  (b)  optimization  in  terms  of  zero  mean 
error  and  minimum  error  variance. 

Simon  [28]  was  the  first  to  introduce  the  cubic  convolution  interpolation 
using  splines  and  formulating  the  resampling  problem  as  a  constrained  linear 
estimation  problem  with  suitable  image  models  and  optimization  criteria.  In 
this  paper  the  interpolation  kernel  is  called  "cubic  convolution,"  but  the 
piecewise  cubic  polynomials  used  for  the  approximation  are  not  simple  splines 
by  nature,  since  they  do  not  follow  the  continuity  criteria  that  are  specific 
to  spline  functions. 

A  spline  function  of  order  k  is  defined  to  be  a  piecewise  polynomial 
function  of  order  k  on  some  interval  with  k-2  continuous  derivatives  [29,30], 
In  other  words,  a  spline  function  is  a  piecewise  polynomial  function  that  is 
as  smooth  as  it  could  be  without  simply  reducing  it  to  a  polynomial. 
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The  one-dimensional  cubic  convolution  kernel  given  by  Simon  is  one  of  the 

form 

k(0)  =  1 

k(±D  =  0 

k(x)  =0,  }x |  >  2 
k (x )  =  k(-x) 

with  the  constraint  equation 

k(x)  +  k(l  +  x)  +  k(l  -  x)  +  k(2  -  x)  =  1 

and  the  further  constraint  of  continuity  of  the  first  derivative  of  all  knot 
points.  The  above  defines  a  cubic  polynomial  with  one  degree  of  freedom, 
which  can  be  taken,  for  example,  such  that  the  derivative  of  k(x)  at  x  =  1  be 
the  same  as  that  of  sine  (irx).  This  gives  the  following  polynomial  as  an 
approximation  to  a  sine  function  at  four  points: 

1  -  2  | x | ^  +  |  x  [ 3  if  0  <  | x |  <  1 

k ( x )  =  4-8  |  x  |  +  5  |  x  | 2  -  |  x  | 3  if  1  <  x  <  2 

0  otherwise 

which  is  a  continuous  function  with  a  continuous  first  derivative  at  equally 
spaced  knot  points,  -1,  0,  1,  2,  but  a  discontinuous  second  derivative  at  +1. 
This  discontinuity  indicates  that  the  above  function  is  not  a  regular  spline 
function  with  all  its  continuity  properties,  but  is  a  sort  of  smoothness 
deficient  spline  [29-31], 

E.  DETAILS  OF  CALCULATION 

In  one-dimensional  cubic  convolution,  four  terms  are  Included  in  the 
summation,  i.e.,  only  four  pixel  values  (u^-j,  u^,  u^+j  and  u|+2^  nearest  to 
the  desired  location  (X)  contribute,  and  the  interpolation  equation  becomes 


i+2 

g(x)  =  l  f (u  )  k(x-u  ) 

1-1  1  • 

It  is  reasonable  to  assume  that  each  array  of  raw  image  data  along-scan  has 
evenly  distributed  pixels  with  spacing  -  u^  =  h.  Thus,  the  above 

equation  reduces  to 

g(x)  =  f(u^_1)  k(6x+h)  +  f(ui>  k(6x) 

+  f(u^+1)  k ( 6x~h )  +f ( u^ ^2 )  k(6x-2h) 

where  x  =  uj  -  6x  and  h  =  u^  +  i-up 

A  graphical  presentation  of  the  four  k-values  for  the  cubic  convolution 
kernel  described  in  the  previous  section  is  given  in  Figure  17.  The  summation 
in  the  above  equation  is  also  illustrated  in  Figure  18  where  f^  =  f(u^) 

kj  =  k(<5x+t),  k^  =  k(6x),  k^  =  k(6x-h)  and  k^  =  k(6x-2) 

and  the  dot  represents  the  interpolated  value  g(x). 

In  practice,  several  6x  values  are  given;  the  corresponding  k-values  are 
precomputed  and  stored  in  a  vector  to  be  used  during  the  resampling  process. 

In  two-dimension,  the  same  process  can  take  place,  except  K  is  a  matrix 
of  4  x  4  and  16  pixel  values  are  considered  for  interpolation.  However,  as 
described  previously,  the  resampling  process  usually  takes  place  in  two 
stages:  first  in  the  direction  of  raw  data  scan  (along-scan  line),  as  out¬ 

lined  above;  and  then  in  the  orthogonal  direction  (across-scan  lines). 
Across-scan  line  interpolation  proceeds  in  the  same  manner  as  along-scan  line, 
except  interpolation  is  along  columns  of  interpolated  data  (in  the  x-v  coor¬ 
dinate  system).  Final  output  is  usually  interpolated  along-  and  across-scan 
and  is  presented  in  the  x-y  coordinate  system  as  a  regular  grid  of  pixels 
(32-34 J. 
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INTERPOLATED 


ANALYSIS  OF  RESAMPLING  ALGORITHMS 


As  pointed  out  previously,  the  most  commonly  used  resampling  technique 
(replication  or  nearest  neighbor  interpolation)  reduces  the  resolution  of  the 
image,  igiving  it  a  blocked  look  and  changing  diagonal  edges  into  staircase 
responses.  It  produces  errors  of  the  order  of  ±0.5  pixel  spacing  which,  in 
turn,  results  in  some  sort  of  jitter  in  the  high  contrast  regions  of  the 
imagery.  The  familiar  bilinear  interpolation  technique  has  a  smoothing  effect 
that  is  pleasing  to  the  eye  but  causes  degradation  around  edges  and  highlights 
of  imagery.  Cubic  convolution  has  less  of  a  smoothing  effect,  but  it  is 
harder  to  implement. 

Some  resampling  results  [27-35]  suggest  advantages  of  nearest  neighbor 
and  bilinear  interpolation  over  resampling  by  cubic  convolution  as  a  result  of 
overshooting  of  data  and  exaggeration  of  peaks  (valleys)  occurring  in  the 
cubic  convolution. 

However,  from  the  outcome  of  all  analysis  to  date,  it  seems  that  the 
cubic  convolution  method  is  the  leading  method  as  far  as  computing  time, 
statistical,  and  frequency-content  considerations  lie.  On  the  other  hand, 
artifacts  such  as  ringing  or  other  unpleasant  visual  effects  may  still  be 
present  in  the  final  image  after  resampling  with  cubic  convolution.  Note  that 
many  artifacts  seen  in  the  resampled  data  are  independent  of  the  specific 
interpolation  method  and  actually  present  in  the  original  data  as  well.  More 
analysis  of  the  effect  of  resampling  algorithms  on  various  artifacts  such  as 
speckle,  shadowing,  and  contrast  reversals  is  under  active  study. 


VI.  SAR  IMAGE  RECTIFICATION 


At  present,  not  enough  results  and  studies  are  available  on  the  matching 
of  SAR  images  to  visible  image  or  to  a  SAR  reference  map  to  draw  conclusions 
from.  Most  of  the  previous  work  has  been  concentrated  in  the  rectification 
and  registration  of  visible  imagery  as  discussed  in  previous  sections.  This 
section  deals  with  specific  qualities  of  SAR  imagery  which  may  cause  serious 
problems  in  accurate  image  matching  and  investigates  the  feasibilities  of 
various  rectification,  registration,  and  change  detection  algorithms 
applicable  to  SAR  imagery. 

A.  BACKGROUND 

The  specific  qualities  of  SAR  data  must  be  considered  when  a  SAR  image  is 
to  be  registered  with  a  visible  image  or  when  two  SAR  images  are  to  be  regis¬ 
tered.  SAR  imagery  is  generally  different  statistically  and  visually  from 
imagery  obtained  by  visible,  IR,  or  other  wavelength  sensors.  SAR  imagery 
tends  to  have  much  lower  pixel-to-pixel  correlation  than  visible  imagery,  and 
may  have  much  greater  dynamic  range.  For  visible  imagery,  a  dynamic  range  of 
8  bits  (256  to  1 )  is  usually  adequate;  while  for  SAR  data,  the  dynamic  range 
may  exceed  1000  to  1  [36-38],  Bright  specular  reflections  (glints)  from  plane 
surfaces  such  as  roofs  and  strong  returns  from  corners  of  man-made  structures 
contribute  to  this  great  dynamic  range. 

The  coherent  nature  of  SAR  creates  artifacts  such  as  speckle  and  false 
texture  in  optically  diffuse  areas  of  the  scene.  Speckle  is  a  granular  noise 
arising  from  the  random  interference  of  wavefronts  scattered  from  an  optically 
rough  surface  [41,42].  In  its  most  general  form,  speckle  is  signal-dependent 
and  very  objectionable  visually.  Speckle  may  create  the  appearance  of  texture 
in  regions  that  appear  uniformly  bright  at  a  different  wavelength.  The  spatial 
scale  size  of  the  speckle  is  generally  on  the  order  of  the  limiting  resolution 
of  the  coherent  SAR  imaging  system  [41-43]. 

Other  artifacts  include  shadowing  of  surface  relief  features  and  contrast 
reversals  in  comparison  to  visible  imagery. 
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Figure  19  is  a  simulation  showing  how  the  appearance  of  an  aerial  photo¬ 
graph  changes  with  visible  and  SAR  sensors  [41].  Figure  19(a)  is  an  example 
of  an  aerial  photo  at  visible  wavelengths.  Figure  19(b)  shows  a  digitally 
simulated  coherent  image  with  speckle  simulated  using  an  exact  signal- 
dependent  model.  It  has  a  signal-to-noise  ratio  of  2.  The  simulation  does 
not  include  some  particular  SAR  artifacts  such  as  shadowing,  specular 
reflections,  and  contrast  inversions,  but  does  include  granular  speckle  noise. 

Various  techniques  for  the  suppression  of  speckle  have  been  investigated 
[41,  42,  44].  One  class  of  techniques,  which  increases  the  signal-to-noise 
ratio  by  incoherently  averaging  independent  looks,  can  be  obtained  by  polari¬ 
zation,  wavelength,  and  spatial  frequency  diversity.  Other  techniques  use 
recursive  and  nonrecursive  image  restoration  to  spatially  smooth  speckle  noise 
while  preserving  object  details.  Figures  19(c)  and  19(d)  show  examples  of 
recursive  filtering  applied  to  images  with  speckle.  Figure  19(c)  shows  the 
image  of  Figure  19(b)  processed  by  a  local  linear  minimum  mean-square  error 
filter;  Figure  19(d)  shows  Figure  19(b)  processed  by  a  maximum  a  posteriori 
(MAP)  filter. 

Very  little  theoretical  and/or  experimental  work  on  the  matching  of  SAR 
images  Is  available  [17,48,50-55].  Referring  to  Figures  3,  20,  and  21, 
describing  image  matching  methods,  it  is  likely  that  signal-processing-based 
correlation  techniques  applied  to  SAR  pixel  data  would  be  the  most  effective. 
These  techniques  provide  some  noise  immunity  by  their  inherent  spatial  averag¬ 
ing  over  the  window  area.  It  is  probable  that  the  usual  correlation  computa¬ 
tion  over  windows  would  work  well  on  SAR  data,  although  it  Is  unclear  how 
more-sophisticated,  fast  algorithms  such  as  sequential  similarity  detection 
(SSD),  which  do  not  use  all  data  in  the  window,  would  work. 

The  artificial  intelligence  (Al)-based  techniques  shown  in  Figures  20  and 
21  generally  require  clean,  noise-free  image  data,  and  robust  features  such  as 
edges,  boundaries,  and  vertices.  The  Al-based  matching  problem  then  reduces 
to  a  symbolic  matching  of  directed  graphs  or  other  relational  structures  using 
relaxation  or  other  techniques.  Hybrid  techniques  are  useful  in  that  they  may 


A  =  ORIGINAL  VISIBLE  IMAGE 


B  -  SIMULATED  COHERENT  IMAGE 

4  FRAME  AVERAGE  OF  INDEPENDENT  SPECKLE  IMAGES.  S/N  =  2 

C  =  IMAGE  B  PROCESSED  BY  LOCAL  LINEAR  MINIMUM  MEAN-SQUARE 
ERROR  FILTER 

D  =  IMAGE  B  PROCESSED  BY  MAP  (maximum  a  posteriori 
probability!  FILTER 


Figure  19.  Visible  and  Synthetic  SAR  Images 
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Figure  20.  SAR  Image  Matching  Techniques 
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provide  some  spatial  averaging,  but  good  extraction  of  lines,  edges,  and  other 
linear  features  is  still  required. 


Much  research  is  still  needed  on  the  matching  of  SAR  imagery.  One 
approach  is  to  investigate  various  preprocessing  algorithms  for  noise  removal, 
followed  by  matching  techniques  suitable  for  visible  data  (either  signal 
processing  or  AI ) .  Another  approach  may  be  to  segment  a  SAR  scene  into 
regions  of  common  texture  and/or  brightness,  extract  outlines  of  the  segmented 
regions,  and  combine  this  with  other  edge/line  data  derived  from  the  scene  as 
input  to  a  matching  procedure.  For  AI -based  symbolic  matching,  more  a  priori 
knowledge  specific  to  SAR  can  be  included.  For  example,  in  a  visible  image 
Al-based  matching  system,  knowledge  that  bodies  of  water  generally  appear 
darker  than  large  land  masses  can  be  used  in  matching.  This  fact  may  not  be 
true  in  SAR  imagery,  and  a  knowledge  base  of  relevant  facts  such  as  these  can 
be  developed  and  incorporated  into  the  matching. 

A  related  problem  concerns  the  matching  of  SAR  images  to  visible  images. 
In  this  case,  significant  contrast  reversals  between  the  scenes  will  occur, 
thus  hybrid  or  feature  matching  techniques  may  be  the  only  ones  that  work 
[48]. 

S.  SAR  IMAGE  MATCHING  ALGORITHMS 

Very  little  work  is  available  on  the  matching  of  SAR  images  to  visible 
reference  maps  or  SAR  images.  As  discussed  earlier,  most  previous  work  has 
been  concerned  with  the  matching  of  visible  images  to  visible  images.  The 
introduction  of  speckle  scintillation  by  SAR  systems,  along  with  magnifica¬ 
tion,  rotation,  geometrical  distortion,  and  other  sources  of  mismatch  makes 
the  general  SAR  matching  problem  very  difficult.  This  section  reviews  the  few 
techniques  of  matching  that  specifically  consider  SAR  images. 

Novak  [57,58]  has  studied  a  generalized  correlation  approach  to  the 
matching  of  map  reference  data  to  SAR  data.  He  specifically  considers  the 
effects  of  speckle  and  the  reduction  of  processing  complexity  using  fast 
implementation  techniques.  Novak  uses  correlation  with  radar  edge  features 
(for  example,  land/water  contours,  tree/field  contours)  that  have  sufficient 


radar  contrast,  unique  shape,  and  length  to  provide  reasonable  robust  perfor¬ 
mance.  The  actual  matching  is  done  with  bipolar  (+1  or  -1)  edge  or  shape 
templates  such  as  shown  in  Figure  22.  The  unit  weights  of  the  template  mean 
that  multiplies  are  not  required,  thus  reducing  the  computational  complexity. 
The  template  is  correlated  with  the  input  SAR  image,  and  peaks  that  exceed  a 
threshold  are  recorded  along  with  translational  information.  For  all  correla¬ 
tion  peaks  above  a  detection  threshold,  a  shape  or  fill  factor  equal  to  the 
number  of  pixels  within  the  +1  template  whose  value  lies  above  an  average  of 
the  template  sum  is  computed.  After  a  search  over  the  whole  image,  the  corre¬ 
lation  peak  with  the  largest  fill  factor  is  chosen  as  the  best  match  point. 

Several  generalizations  of  the  basic  correlation  method  are  given  by 
Novak  [57],  each  having  different  techniques  of  computing  a  correlation 
measure.  The  effect  of  scintillation  and/or  speckle  is  minimized  by  the 
averaging  of  many  resolution  cells  on  each  side  of  the  edge  template  as  part 
of  the  correlation.  For  one  particular  correlation  measure  (called  "edge 
ratio"),  Novak  derives  statistical  estimates  of  the  probability  of  feature 
detection  and  false  alarm  based  on  edge  contrast  ratio,  edge  length,  edge 
shape,  and  template  width.  He  also  gives  an  experimental  verification  of  per¬ 
formance  using  real  SAR  data.  In  another  paper  [20]  Novak  extends  the  pro¬ 
cedure  to  allow  for  adaptive  detection  thresholds  (in  effect  assuming  a 
spatially  nonstationary  image  model)  and  considers  effects  of  clutter  from 
other  targets,  decoys,  or  specular  reflections. 

Hiller  [55]  has  described  a  shape  matching  approach  for  SAR  image 
matching.  Like  the  techniques  described  by  Novak,  Hiller's  method  has  little 
dependence  on  absolute  intensity  and  contrast  of  the  scene.  He  uses  reference 
binary  templates  like  that  in  Figure  22  that  are  correlated  with  the  input 
scenes  and  exploits  the  binary  characteristics  in  various  ways  to  achieve 
computational  savings.  The  inherent  averaging  provides  resistance  to  scene 
scintillation  and  speckle.  The  correlation  results  are  compared  to  an  adap¬ 
tive  threshold  computed  locally  over  the  scene.  The  adaptive  threshold  is 
effectively  self-normal izing  and  reduces  the  effects  of  intensity  and  contrast 
variations.  Hiller  derives  theoretical  results  on  the  expected  correlation 
value  and  probability  of  feature  detection  as  a  function  of  shape  parameters. 


1  TEMPLATE 


LAND  AREA 


Figure  22.  Bipolar  Edge  Template  for  Image  Matching 


His  theory  provides  a  good  match  with  experimental  results  obtained  by  corre¬ 
lation  over  a  large  ensemble  of  SAR  images. 

Kiremidjian  has  described  in  several  papers  [4-6]  image-to-map  registra¬ 
tion  algorithms  that  specifically  consider  problems  of  SAR.  He  is  concerned 
with  the  registration  of  2-D  reconnaissance  SAR  images  to  a  3-D  reference. 

His  presentation  is  general  and  is  followed  later  by  a  discussion  of  the 
problems  of  tactical  strip  map  SAR  image  matching.  Kiremidjian  's  techniques 
involve  the  computation  of  correspondences  between  a  2-D  SAR  image  model  and  a 
3-D  map  data  base  in  terms  of  a  parameter  set  such  as  altitude,  range,  and 
scale.  If  the  exact  values  of  the  parameter  set  were  known,  the  model  then 
predicts  the  2-D  SAR  image  coordinates  for  any  3-D  data  base  point,  and 
registration  is  achieved.  However,  ephemeris  data  from  the  sensor  platform 
generally  gives  estimates  of  the  model  parameters.  The  goals  of  Kiremidjian 's 
techniques  are  to  refine  these  estimates  so  that  the  location  of  any  data  base 
point  can  be  found  in  the  sensed  image  within  some  desired  accuracy. 
Kiremidjian  then  discusses  real-time  or  near  real-time  methods  for  achieving 
these  objectives  in  a  tactical  setting. 

Two  major  techniques  are  given  by  Kiremidjian.  The  first  [39]  requires 
human  operator  involvement  to  pinpoint  control  point  locations  in  the  sensed 
images.  The  model  predictions  of  these  same  control  points  in  the  map  data 
base  are  then  compared  to  the  control  points  obtained  by  human  involvement, 
and  the  model  parameters  are  searched  for  sets  of  values  that  minimize  the 
distances  between  pairs  of  points. 

The  second  technique  [54,59,60]  is  more  automated  and  relies  on  the  fact 
that  landmarks  such  as  land-water  boundaries,  roads,  and  fences  produce  edges 
in  SAR  data.  In  this  procedure,  the  locations  of  landmarks  in  the  data  base 
are  matched  to  edges  extracted  from  the  SAR  image.  A  search  over  the  model 
parameters  gives  the  best  match. 

For  both  techniques,  Kiremidjian  gives  details  of  the  geometric  model  for 
Image-to-map  registration  and  describes  several  algorithms  and  techniques  for 
parameter  refinement  and  searching.  He  includes  experimental  results  for  both 


techniques  that  prove  the  differences  of  the  methods.  His  later  papers 
[54,60]  include  details  of  edge  extraction,  noise  suppression  and  thinning  on 
the  SAR  image  data,  and  contain  a  detailed  discussion  of  applications  to 
strip-map  SAR  imagery  with  resolution  of  512  x  4096  pixels.  He  demonstrates  a 
location  accuracy  of  50  m  for  this  type  of  system.  Kiremidjian  's  work  is  one 
of  the  most  complete  and  relevant  references  available. 

Another  study  made  using  assumptions  similar  to  Kiremidjian  has  been 
reported  by  Naraghi  and  Stromberg  [48].  They  are  also  concerned  with  the 
matching  of  SAR  imagery  to  a  digital  3-D  topographic  data  base,  and  matching 
to  previously  acquired  SAR  images.  One  of  their  techniques  assumes  that  a 
known  set  of  matchpoints  is  extracted  by  human  interaction,  and  these  tie- 
points  alone  are  used  to  determine  the  SAR  image  distortion.  They  include  a 
discussion  of  geometric  correction  (rubber  sheeting)  by  polynomial  matching 
functions  and  a  mathematical  analysis  of  the  geometry  connecting  2-D  images 
and  3-D  topographic  data  bases.  Another  technique  given  by  Naraghi  and 
Stromberg  is  similar  to  K i remid j ian ' s  in  that  platform  ephemeris  data  are 
combined  with  tiepoint  information  for  registration.  The  context  of  Naraghi 
and  Stromberg's  discussion  is  somewhat  different  because  they  concentrate  on 
Seasat  SAR  data  analysis. 


VII.  PROPOSED  NEW  TASKS  FOR  SAR  IMAGERY 


The  first  phase  of  the  study  objective  was  to  investigate  and  analyze  the 
general  process  of  rectlf Lcation.  However,  in  the  second  phase,  the  emphasis 
was  placed  mostly  on  the  SAR  image  matching.  Part  of  the  objective  of  the 
second  phase  was  to  propose  and  develop  techniques  for  registration  and  change 
detection  with  SAR  images.  These  techniques  may  include  SAR  image  matching  to 
other  SAR  reference  images,  to  visible  references,  or  to  FLIR  reference 
images.  Because  the  problem  is  generally  quite  difficult  and  little  work  has 
been  previously  done  on  the  problem,  several  approaches  are  given  in  the 
following  sections.  The  results  generally  apply  to  both  registration  (finding 
of  match  points  for  spatial  warping)  and  to  change  detection  once  Images  have 
been  registered.  These  problems  are  highly  interrelated  and  differ  mainly  in 
the  window  size  over  which  the  matching  must  be  performed. 

Based  on  the  analysis  In  the  areas  of  signal  processing  and  artificial 
intelligence,  four  specific  tasks  are  proposed  for  image  registration/recti¬ 
fication  and  possible  change  detection  of  SAR  data.  The  first  task  involves 
registration  through  geometrical  transformation  and  resampling.  The  second 
task,  which  requires  considerable  research  effort,  involves  total  registration 
and  change  detection  through  a  combination  of  signal  processing  and  artificial 
intelligence  techniques.  The  third  task  is  basically  a  simplified  version  of 
the  second  task  for  registration  of  SAR  images  with  a  known,  well-defined 
visible  image. 

The  last  task  involves  man-machine  interactive  methods  for  image  rectifi¬ 
cation.  System  hardware/software  requirements  and  configurations  are  also 
discussed  for  this  task.  This  approach  to  rectification  allows  for  analytical 
evaluation  and  comparison  of  various  registration  algorithms  and  their  perfor¬ 
mance  in  addition  to  the  utilization  of  a  photointerpreter's  inputs  to  the 
system. 


TASK  1  ~  REGISTRATION  THROUGH  TRANSFORMATION 
AND  RESAMPLING 


The  task  objective  is  to  develop  techniques  for  registration  of  SAR 
imagery  in  cases  where  a  reference  image  may  not  be  available  (Figure  23). 
Input  may  consist  of  the  digital  sensed  image,  some  geometric  preflight 
parameters,  and  calibration  data  (i.e.,  geometric  control  information). 

Control  information  may  include  scanner  location  and  pointing,  ground  truth, 
and  map  transformation. 

The  method  of  approach  proposed  here  for  digital  registration/rectifica¬ 
tion,  consists  of  two  major  steps.  First,  a  rectified  model  is  computed  from 
the  control  information  utilizing  transformation  techniques.  Second,  the 
image  is  warped  according  to  the  rectified  model  through  application  of 
resampling  and  interpolation  techniques.  For  large  images,  highly  specialized 
computational  methods  for  efficient  warping  and  complicated  data  processing 
for  error  optimization  is  required.  Smaller  images  or  subimages  can  be 
handled  in  a  much  simpler  way.  With  more  preflight  and  inflight  information, 
higher  degree  geometrical  warping  polynomials  can  be  used,  and  as  a  result, 
more  accurate  registration  is  expected. 

Mathematically,  geometrical  distortions  are  mappings  from  plane  to  plane 
that  specify  where  a  point  in  one  domain  appears  in  another  domain.  Section 
III  of  this  report  gives  details  about  the  properties  of  distortion  functions 
and  techniques  for  obtaining  them  in  various  applications .  Some  of  the  tech¬ 
niques  described  are:  polynomial  fit,  nearest-neighbor  interpolation,  finite 
element  method,  and  the  method  of  potential  functions.  These  are  used  in 
cases  where  there  is  no  need  or  desire  to  use  a  priori  knowledge  of  the  nature 
of  the  geometric  distortion.  If  one  knows  (from  physical  considerations)  the 
general  functional  form  of  a  distortion,  then  there  are  methods  (least 
squares,  Kalman  filtering,  etc.)  of  fitting  the  functional  form  to  the 
observat  tons . 

The  most  complex  distortion  models  arise  from  sensor  geometry  correction. 
Taking  the  Landsat  mult ispectral  scanner  (MSS)  as  a  basic  example,  the  raw 
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data  are  perturbed  by  earth  rotation,  mirror  scan  nonl Lnearl ty,  spherical 
earth,  variations  in  platform  altitude,  roll,  pitch  and  yaw.  Most  of  these 
components  can  be  modeled  by  continuous  functions,  but  one  component,  the 
line-to-line  skew  induced  by  earth  rotation,  is  discontinuous  at  every  sixth 
line.  Furthermore,  the  distortion  model  is  no  longer  a  simple  function  but  a 
composite  of  several  functions  that  are  applied  in  order.  The  first  correc¬ 
tion  function  calculates  uniform  sample  spacing  in  orthographic  or  Mercator 
projection  along  single  scan  lines.  Then  a  second  correction  function  moves 
entire  lines  according  to  the  sensor  and  earth  rotation  skew  for  each  line.  A 
third  correction  for  map  projection  could  now  be  performed  if  desired. 

Two  basic  techniques  for  model  fitting  are  in  common  use  today.  The 
first  is  to  use  nominal  values  for  spacecraft  location,  etc.,  and  produce  a 
correlated  product  which  has  slight  deviations  from  a  perfectly  mapped 
product.  Note  that  the  largest  deviation  is  a  simple  lateral  translation, 
which  can  be  fixed  later  by  a  single  point  observation.  The  second  technique 
is  to  fit  the  model  according  to  control  points  determined  by  external  means. 
For  our  purpose,  in  which  a  control  point  (or  reference  map)  may  not  be 
available,  the  first  technique  is  preferable. 

Digital  computation  requires  that  the  geometric  distortion  be  represented 
in  an  efficient  manner.  The  method  proposed  here  is  to  convert  the  rectified 
model  to  a  gridded  approximation  through  the  use  of  resampling  methods.  There 
are  approximately  three  techniques  commonly  used  in  resampling  of  one-  or 
two-dimensional  data;  nearest  neighbor,  bilinear  interpolation,  and  cubic 
convolution.  These  techniques  are  reviewed  in  detail  in  section  V. 

The  above  registration/rectification  approach  requires  both  analytical 
and  computer  simulated  modeling.  The  simplified  version  (affine  transforma¬ 
tion  plus  resampling)  can  be  constructed  in  four  months'  time,  as  a  first  step 
toward  a  more  complete  modeling. 

B.  TASK  2  -  REGISTRATION  AND  CHANGE  DETECTION  OF  SAR  IMAGES 

The  task  objective  is  to  develop  techniques  for  registration  and  change 
detection  with  SAR  images.  These  techniques  may  include  SAR  image  matching  to 
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other  SAR  reference  images,  to  visible  references,  or  to  FLIR  reference 
images.  Because  the  problem  is  generally  quite  difficult  and  little  work  has 
been  previously  done  on  the  problem,  several  approaches  are  proposed.  The 
results  generally  apply  to  both  registration  and  to  change  detection  once 
images  have  been  registered. 

Two  major  classes  of  techniques  are  proposed.  The  techniques,  called  A 
and  B,  share  common  techniques  in  matching  and  differ  mainly  in  the  feature 
extraction  and  preprocessing.  Because  the  general  matching  problem  is  so 
difficult  and  has  such  a  broad  application,  specialized  techniques  suitable 
for  a  restricted  class  of  situations  may  be  more  effective.  If  specific 
knowledge  about  the  scope  of  images  to  be  encountered  is  available,  the 
general  solution  may  be  greatly  simplified. 

Either  of  these  techniques  deserves  study  and  consideration,  but  both  are 
not  well-defined  and  require  considerable  research  effort.  They  require 
significant  experimental  work,  including  computer  time  and  the  services  of  a 
full-time  applications  programmer  in  addition  to  the  members  of  the  technical 
staff  who  will  direct  the  research. 

1 .  TECHNIQUE  A 

This  technique  of  registration  is  outlined  briefly  in  Figure  24.  SAR 
images  are  acquired  and  smoothed  and/or  filtered  by  recursive  or  nonrecursive 
image  restoration  methods  [44-48].  Then  edge  and  texture  features  are 
extracted.  The  edges  may  be  linked  and  expressed  as  a  line  segment  data  base, 
and  the  texture  features  may  be  obtained  by  convolution  with  a  set  of  small 
window  microtexture  kernels  [49].  These  features  then  go  to  a  segmentation 
routine  that  breaks  the  image  into  regions  of  similar  characteristics.  The 
shapes  of  the  segmented  regions  are  extracted  and  form  the  input  to  the  match¬ 
ing  phase  of  the  system.  A  similar  or  identical  set  of  operations  is  per¬ 
formed  on  the  reference  image. 

The  second  phase  of  Technique  A  is  matching.  Several  major  methods  are 
listed;  in  reality  a  hybrid  approach  combining  some  of  them  may  be  used.  The 
method  performs  shape  matching  of  lines,  linear  features,  vertices,  etc.,  that 


are  stored  in  a  data  base.  Similar  methods  have  been  explored  by  Hughes 
Research  Laboratories  [17,  51-55]  and  others.  Another  method  is  to  match 
symbolic  descriptions  of  images,  in  which  the  structure  is  represented  by 
lines,  vertices,  higher  order  shape  descriptions,  and  texture.  Techniques 
such  as  relaxation  labeling/matching  are  used  to  find  the  best  match.  For 
certain  restricted  problems,  the  signal -processing-based  techniques  of  corre¬ 
lation  matching  may  be  effective  and  offer  the  inherent  advantage  of  noise 
reduction.  This  method  may  be  useful  for  matching  with  a  well-defined  set  of 
reference  Images  or  objects  within  an  image,  but  will  generally  fail  with 
significant  changes  between  reference  and  image  to  be  matched. 

The  entire  approach  of  Technique  A  follows  and  expands  on  previously 
demonstrated  techniques  used  for  visible-to-visible  image  matching.  It  is 
hoped  that  the  use  of  noise  suppression  combined  with  textural  information 
will  be  sufficient  for  robust  performance  even  though  SAR  images  are  signifi¬ 
cantly  noisy  and  different  statistically  and  visually  from  visible  references. 

2.  TECHNIQUE  B 

This  technique  differs  from  A  in  the  preprocessing  steps.  SAR  images  are 
acquired  and  edge  detection/linking  is  performed  as  before.  Texture  feature 
extraction  is  performed,  but  over  a  hierarchy  of  several  spatial  resolutions 
and  window  sizes  (Figure  25).  This  textural  information  is  input  to  a  segmen¬ 
tation  method  that  attempts  to  find  texture  boundaries  in  the  scene.  The 
texture  segmentation  is  done  by  unsupervised  clustering  over  the  hierarchy  of 
mult i resolut ion  data.  This  method  is  motivated  by  knowledge  of  the  human 
visual  system,  which  uses  spatial  detectors  and  sensory  information  over  a 
range  of  spatial  resolution  in  order  to  discriminate  texture  boundaries.  This 
area  of  study  is  the  subject  of  current  research  by  several  groups  [57], 

Following  texture  segmentation,  shapes  are  extracted  and  input  to  the 
various  shape  matching  procedures  described  before.  Reference  images 
(visible,  SAR,  FLIR,  etc.)  are  processed  by  a  similar  sequence  (although  less 
texture  information  may  be  needed  for  segmenting  visible  data). 
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TASK  3  -  MATCHING  SAR  IMAGES  TO  VISIBLE  IMAGES 


This  task  is  essentially  a  simplified  version  of  Task  2  for  the  registra¬ 
tion  and  matching  of  SAR  images  acquired  in  real-time  to  'a  stored  visible  data 
base . 

The  method  of  approach  is  basically  similar  to  the  two  techniques 
outlined  in  Task  2,  except  that  we  assume  the  existence  of  a  good  quality 
visible  image  data-base  to  which  the  SAR  data  are  matched.  The  visible  data 
can  consist  of  stored  imagery,  extracted  features  such  as  lines,  edges, 
segmented  regions,  and  ground  control  points.  Other  data,  such  as  previously 
acquired  graphics,  roads,  physical  structures,  or  map  data  can  be  included. 
Similar  features  are  extracted  from  the  real-time  SAR  input  and  are  matched  as 
described  in  Task  2.  Following  the  matching,  spatial  warping  and  registration 
can  be  performed  as  described  in  Task  1  . 

The  advantages  of  this  task  as  compared  with  Task  2  is  that  this  is  a 
much  simpler  problem  that  can  be  accomplished  with  fewer  personnel  in  a 
shorter  amount  of  time. 

D.  TASK  4  -  MAN-MACHINE  INTERACTIVE  METHOD 

This  section  discusses  proposed  man-machine  interactive  methods  for  Image 
rectification,  system  hardware/software  requirements,  and  configurations. 

The  proposed  procedure  allows  for  analytical  evaluation  and  comparison  of 
various  registration  algorithms  and  their  performance  for  the  specific  generic 
system  under  consideration.  In  addition,  the  new  procedure  utilizes  the 
inputs  of  a  photointerpreter  (PI)  and  feedback  to  the  system  in  such  a  way  as 
to  minimize  computer  storage  and  computing  time  requirements. 

Most  of  the  rectification  methods  used  in  the  past  ignore  the  man-machine 
interaction  aspect  of  the  problem  or  try  to  make  a  fully  automated  rectifica¬ 
tion  process.  However,  utilization  of  the  PI  In  the  loop  may  allow  for  more 
efficient  Image  registration  because  the  ground  truth  and  specific  target 
areas  can  be  identified  in  a  simpler  fashion.  The  following  proposed  approach 
to  rectification  allows  for  analytical  evaluation  and  comparison  of  various 


registration  algorithms  and  their  performance  in  addition  to  the  utilization 
of  a  Pi's  inputs  to  the  system. 

1.  METHODS  OF  APPROACH 

Onboard  sensor  data  in  the  form  of  inflight  or  preflight  information  are 
available  to  the  various  geometrical  rectification/registration  algorithms. 
These  data  may  include  limited  a  priori  knowledge  of  fixed  system  geometrical 
distortions  or  onboard  sensor  attitude  data.  In  the  new  approach  proposed 
here,  general  areas  of  interest  for  magnification,  scale  change,  comparison 
for  change  detection,  etc.,  are  interactively  selected  from  analog  hardcopy 
images  by  the  PI  and  used  as  input  to  the  rectification  algorithm.  Also, 
control  point  locations  corresponding  to  known  areas  of  the  scene  and  other 
points  of  interest  are  selected  interactively  by  the  PI  and  used  in  rectifi¬ 
cation  algorithms. 

Rectification  algorithms  proposed  here  produce  geometric  corrections 
based  on  existing  orientation  data  and  identifiable  ground  control  points 
furnished  by  the  PI.  Information  on  the  ground  control  points,  furnished  by 
the  PI,  could  be  used  in  two  different  fashions:  either  through  parametric 
methods  or  nonparametric  methods. 

The  following  is  a  brief  description  of  the  man-machine  interactive 
methods  proposed  for  the  specific  system  under  study.  It  must  be  noted  that 
in  practice  these  methods  are  applied  only  to  a  coarse  grid  of  points  distri¬ 
buted  through  the  image.  The  remaining  samples  are  obtained  using  resampling 
and  interpolation  procedures  for  greater  efficiency. 

a.  Orientation  Method 

In  spacecraft-borne  sensors,  the  orientation  parameters  are  continuously 
varying  functions  of  time.  Thus,  to  perform  good  registration  of  the  images, 
it  is  necessary  to  know  these  functions  with  sufficient  accuracy.  This  may  be 
accomplished  in  the  following  manner:  approximation,  platform  stabilization 
(costly),  interpolation,  or  geometric  modeling.  Geometric  modeling  through 
the  collinearity  condition  is  proposed  here  to  accomplish  sufficient  accuracy. 
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The  orientation  method  could  use  either  the  preflight  data  or  onboard 
flight  data  which  are  recorded  at  the  same  time  with  the  sensed  images  to 
generate  a  new  image  with  all  distortions  due  to  aircraft  motion  removed. 

These  distortions  include  the  roll,  pitch,  yaw,  and  translational  movements  of 
the  platform. 

b.  Parametric  Methods 

Parametric  methods  in  rectification  make  use  of  a  polynomial  approxima¬ 
tion  to  relate  the  original  and  corrected  image  coordinates.  The  ground 
coordinates  of  some  identifiable  points  in  the  image  (ground  control  points) 
furnished  by  the  PI,  combined  with  least-squares  fitting  techniques,  produce 
the  coefficients  of  the  approximating  polynomial. 

c.  Nonparamet ric  Method 

The  nonparametric  method  of  rectif ication  is  based  on  the  assumption  that 
the  difference  in  coordinates  between  the  original  image  and  the  corrected 
image  is  a  realization  of  a  two-dimensional  stochastic  field.  A  simplified 
version  of  this  approach  is  proposed  here  through  an  arithmetic-mean  tech¬ 
nique,  which  for  each  point  computes  the  difference  in  coordinates  between  the 
original  and  the  corrected  image  as  a  weighted  average  of  those  differences  in 
a  set  of  ground  control  points  identified  by  the  PI. 

2.  HARDWARE  CONFIGURATION 

As  noted  earlier,  the  major  problem  with  Images  presented  to  the  PI  is 
that  they  differ  in  magnification,  orientation,  contrast,  etc.,  due  to  varia¬ 
tions  in  flight  path,  sensor  response,  and  ephemeris  data.  A  tactical,  ground 
based  image  warp  correction  and  registration  system  with  real-time  or  non 
real-time  speed  is  needed  to  implement  the  correction. 

Figure  26  shows  a  possible  system  hardware  configuration.  Figure  26(a) 
is  the  part  of  the  system  assumed  to  be  conventional  SAR  processing.  Figure 
26(b)  shows  the  proposed  additional  parts  of  the  system  that  accomplish  the 
necessary  rectification/registration  of  the  Imagery  selected  by  the  PI. 

Figure  26(a)  shows  a  block  diagram  of  a  typical  SAR  processing  hardware 
configuration.  The  sensor  system  collects  data  and  stores  it  in  analog  form 
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in  a  raw  data  buffer.  For  SAR  systems,  these  data  may  be  stored  optically  on 
photographic  film  or  on  magnetic  tape.  An  analog  SAR  processor  performs 
correlation  and  filtering  of  the  data,  giving  a  hardcopy  output  with  many 
picture  elements  (as  many  as  500  *  500  pixels).  A  PI  views  the  analog  high 
resolution  image  and  identifies  objects  of  interest  and  changes  as  compared 
with  previous  images  of  the  same  general  scene. 

Figure  26(b)  shows  the  remaining  part  of  the  proposed  system  utilized  for 
image  rectification.  Most  of  the  actual  correction  is  implemented  digitally. 
An  analog/digital  converter  (TV,  a  CCD  array,  or  similar  device)  is  used  to 
digitize  analog  image  outputs  and  store  them  in  a  digital  image  buffer.  The 
buffer  contains  sufficient  memory  to  store  a  few  images  (on  the  order  of  100) 
at  the  resolution  needed  for  comparison  with  new  or  existing  data  by  the  PI. 
The  buffer  may  consist  of  fast  access  RAM  (random  access  memory)  or  slower  but 
higher  capacity  disc  or  tape  digital  storage.  The  buffer  also  incorporates  a 
computer  for  implementing  the  correction  and  keeping  track  of  the  data. 

The  image  buffer/computer  uses  the  selected  data  provided  by  the  PI 
(utilizing  analog  hardcopy  images)  to  establish  control  points  in  the 
digitized  digital  image  data  and  fit  geometrical  correction  polynomials  to 
control  point  locations.  Interpolation  by  bilinear  or  other  algorithms  is 
used  to  compute  correction  digital  images  between  the  control  points.  The 
result  is  a  digital  hardcopy  output  of  selected  areas  of  the  analog  scene 
generally  with  fewer  pixels  (on  the  order  of  256  x  256)  than  the  analog  scene. 
The  digital  output  is  produced  in  a  form  easily  overlaid  and  compared  with 
previously  corrected  images. 
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